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September 1988 

TO TOE PRESIDENT AND THE CONGRESS OF THE UNITED STATES 

The National Commission for Employment Policy (NCEP) is pleased to present to you this 
Executive Summary of our major research study entitied. Evaluation of the Effect ofJTPA Per- 
formance Standards on Clients, Services, and Cost. This is the first national evaluation of a 
cornerstone of the Job Training Partnership Act (JTPA) — the implementation of clear, 
measurable program goals that exemplify '*the return on investment'' strategy upon which the en- 
tile program is built In fact, Congress itself believed that performance standaixls were of such 
importance that it mandated an evaluation of their impact in the enabling legislation. Section 
106(f)(2) of the Act directs the Conmiission to ^^evaluate the impact of such standards (intended 
or otherwise) on the choice of who is served, what services are provided, and the cost of such 
service in service delivery areas." 

I believe this report is one of the major contributions of my tenure as Chairman of the Com- 
mission. Contractors working on the study, SRI International and Berkeley Planning Associates, 
are to be commended for their superb job of gathering and analyzing the diverse and compli* 
cated data* As one of our reviewers notes '*this is a tremendously valuable piece of work, break- 
ing new ground in national studies.'' I am confident you will agree that the findings detailed are 
very useful, having implications that go well beyond the performance management system and 
beyond JTPA. Many of the findings can be helpful when enacting legislations for other human 
resource programs. 

The Commission believes that programmatic performance standards have been a successful 
strategy to focus the management of JTPA toward the goals of the legislation— finding produc- 
tive employment for disadvantaged people. This not to say that no adjustments are needed to im- 
prove the performance managemeiit system; however, we feel that, overall, the performance 
management system is having its intended effect of guiding JTPA resources toward meaningful 
investment m the Nation's most precious resource — its human capital. 

The Conunission is pleased to have the opportunity to present you both general and specific 
recommendations that we feel would make the performance management system even more 
valuable to the Nation's largest job training system. Please contact me if I can be of further assis- 
tance to you. 

Sincerely, 

Gertrude C. McDonald 
Chairman 
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PREFACE 

This report is organized into 5 sections with 14 chapters and recommendations. The first sec- 
tion titled "Introduction" includes three chapters containing background material including the 
goals of the evaluation, a description of the conceptual model used, and an overview of the 
Federal and State roles in the performance standards system* 

Section B includes three chapters on the quantitative analysis, which focus on the main is- 
sues of the study— determimng the effect of performance standards on who is served, the types 
of services offered, and the cost of such services. Section C contains six chapters on the qualita- 
tive analysis. Infomiation in this section was obtained from the site visits to States, Service 
Delivery Areas, and service providers. 

Section D contains a summary of the findings and conclusions of SRI International and 
Berkeley Planning Associates, the authors of the report. The final section contains recommenda- 
tions of the National Commission for Employment Policy concerning changes to the perfor- 
mance management system. 

A separate volume contains the Appendixes to the report, including the questionnaires used. 
A separate Executive Summary of the report is also available. Additional copies of the roport are 
available from the National Conmussion for Emplovment Policy, 1522 K Street, NW, Suite 300, 
Washington, DC 20005. 
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I. Introduction and Goals of the Evaluation 



Role of Performance Standards 
in the JTP A Title H- A Program 

Performance standards are an integi.'^l 
part of the Job Training Partnership Act 
(JTPA) programs operated under Title II-A. 
The standards implemented during the first 5 
years of program operation [Transition Year 
1984 through Program Year 1987 (PY 87)] 
were intended to further a variety of Federal 
program priorities. These included 

• To hold service delivery areas 
(SD As) accountable for program 
outcomes, 

• To encourage the achievement of 
quality employment outcomes, 

• To encourage the achievement of cost- 
effective outcomes, 

• To create an incentive for effective 
management by local program ad- 
ministrators, and 

• To foster acceptance of the program 
by the business community. 



The focus on accountability and the con- 
cern with outcome measures — ^rather than 
process measures — were new to employment 
and training programs. The performance 
standards grew out of the legislative goal of 
"measuring the return on the JTPA program 
investment" in terms of increases in the 
employment and earnings of economically 
disadvantaged individuals and reductions in 
welfare dependency. Although not intended 
to be an accurate measure of net program im- 
pacts, the performance standards are used to 
indicate the extent to which individual SDAs 
are managing their resources in order to 
achieve important program outcomes. 

Starting in PY 88, several additional per- 
formance standards have been implemented, 
including measures of the participants* status 
3 montiis after leaving die JTPA program 
The new post-program standards are intended 
to provide greater emphasis on the objective 
of improving job retention and longer term 
employability of participants tiian do die cur- 
rent measures, which measure tiie immediate 
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employment status of participants on leaving 
the program. 

The Federal Role in 
Performance Standards 

Although the performance standards are 
one expression of Federal goals for the JTPA 
programs, the performance-standards system 
has htta care^lly designed to minimize the 
day-to-day Federal role in local program 
direction and administration. The JTPA sys- 
tem is highly decentralized, enabling local 
Private Industry Councils (PICs) and SD As 
to design programs relevant to the needs and 
employment opportunities in their local 
areas. In particular, the local agencies are 
given wide discretion in the types of program 
services provided and the types of clients 
served. 

The Federal role in influencing the 
recruitment and selection of JTPA clients is 
limited to several requirements in the legisla- 
tion: (1) at least 90% of the enroUees be 
economically disadvantaged, (2) recipients 
of Aid to Families with Dependent Children 
(AEDC) and high school dropouts receive 
an equitable share of JTPA services, and (3) 
40% of all JTPA service dollars be expended 
for services to youth. 

Tlie Federal role in determining the types 
of program services provided is even more 
limited, allowing SDAs substantial discretion 
in designing their JTPA service mix. Al- 
though public service employment is not an 
allowable service and expenditures on sup- 
portive services are restricted (limiting the 
ability of SDAs to offer stipends to class- 
room training participants), a broad range of 
services is permitted by the JTPA legislation 
and Federal program regulations. 

Consistent with the limited Federal role 
in designing the JTPA program, the intention 
of Federal performance-standards policies is 
to foster accountability and cost-effective- 
ness without undue influence on SDA design 
decisions. The Federal performance-stand- 
ards policies include the choice of measures. 



the numerical level at which the standards 
are set, and the provision of optional models 
for adjusting standards. 

Careful effort went into choosing Federal 
performance measures that would not con- 
strain SDAs in their decisions about whom to 
serve and what services to offer. The numeri- 
cal level of most Federal standards is set at a 
"minimally acceptable" level so that SDAs 
could generally meet their standards without 
distorting their program designs. The adjust- 
ment models developed for the Federal stand- 
ards are intended to ensure the "neutrality" of 
the standards with respect to decisions about 
client targetmg and to "hold harmless" SDAs 
that choose to serve more difficult clients 
through downward adjustments to the 
numerical levels of their standards. 

The State Role in 
Performance Standards 

States can play a very visible and impor- 
tant role in the JTPA program as a whole and 
in the performance-standards system m par- 
ticular. Specifically, States are responsible 
for setting the levels of standards for SDAs. 
States decide whetiier or not to use tiie 
Department of Labor^s (DOL) adjustment 
model to set SDA standards and make adjust- 
ments beyond tiie model to tiiose standards- 
States may also develop and implement addi- 
tional performance standards to further State 
program goals and priorities and may decide 
how to weight the different State and Federal 
standards in making incentive awards. States 
also determine what portion of the 6% funds 
will be used for technical assistance to 
SDAs, what portion will be used for incen- 
tive awards to SDAs that exceed the stand- 
ards, and what portion will be used for 
incentives to serve the hard to serve. States 
determine whether the incentive awards will 
encourage performance that just exceeds the 
standard level or whether there will be a 
strong financial incentive for SDAs to per- 
form at the highest level possible. Finally, 
States determine policies to sanction SDAs 
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that fail to meet their standards. 

Although the Federal performance-stand- 
ards system was designed to encourage SDA 
discretion about who to serve and what ser- 
vices to offer, State governments are en- 
couraged to use the performance-standards 
system to further State client and service 
priorities. Some States have welcomed the 
opportunity to take a leadership role in these 
areas, while other States have decided to play 
a less active role and defer to the preferences 
of local SDAs. 

The SDA and PIC Role in 
Performance Standards 

SDAs, with guidance from PICs, have the 
responsibility for actually implementing 
JTPA programs and for making major 
program design decisions about service and 
client mix. In making these decisions, SDAs 
try to balance the influence of local goals, 
their assessment of local conditions, the per- 
formance-standards incentives offered by the 
State, and the State and Federal program 
regulations. SDAs are also motivated to per- 
form well on the performance standards for a 
number of reasons, including the desire to (1) 
run cost-effective programs that achieve high- 
quality outcomes, (2) appear to be account- 
able to local elected officials and private 
sector representatives on the PIC, (3) avoid 
reorganization as a consequence of failing to 
perform at acceptable levels for 2 years in 
succession, and (4) receive incentive awards. 

SDA and PICs may respond to perfor- 
mance-standards incentives in three key areas 
of program design: (1) decisions about the 
types of clients to serve and procedures to 
recruit and enroll clients; (2) decisions about 
the types of services to provide and the 
length of those services; and (3) decisions 
about how to provide services, including the 
choices of service providers, type of contract, 
and contiact terms. 



Objectives of the Study and 
Key Policy Questions 

The National Commission for Employ- 
ment Policy (NCEP) has funded a com- 
prehensive evaluation of several important 
issues, in keeping with its legislative mandate 
to advise the Secretary of Labor on the 
development of the JTPA performance stand- 
ards and to evaluate the impact of standards 
on the choice of who is served, what services 
are provided, and the cost of such services at 
the local SDA level (Section 106 (f)). The 
overall goal of the study is to assess whether 
the performance standards have influenced 
SDA and service provider decisions about 
program design and implementation practices 
in ways that have affected the clients served, 
the types of services provided, and the costs 
of the services. 

The study is not a detailed assessment 
of whether the performance standards have 
resulted in an increase in JTPA program 
productivity and cost-effectiveness. Rather, 
the study^s purpose is to test whether the in- 
centives created by the performance-stand- 
ards system have caused SDAs to emphasize 
measured performance objectives at the ex- 
pense of unmeasured or unrewarded objec- 
tives that are also valued. In particular, the 
goal is to determine whether emphasis on the 
performance standards as a vehicle for 
program management has led SDAs to avoid 
serving more difficult clients or to reduce the 
intensity of services offered, which in turn 
may have affected the types of participants 
who could benefit from the program. 

The evaluation was intended both to 
determine the effects of performance-stand- 
ards policies and to investigate how those ef- 
fects come about Thus, the evaluation was 
designed to meet several goals. First, the 
study was designed to assess the effect of the 
overall Federal performance-standards sys- 
tem on SDA and service-provider behavior. 
Our examination of the Federal performance- 
standards system distinguished among the 
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effects of (1) the kinds of performance 
measures selected, (2) the level at which per- 
formance expectations were set, and (3) the 
adequacy of the adjustment model in holding 
SDAs "harmless" for their client and service 
choices. 

Second, the study was designed to assess 
how various State performance-standards 
policies and procedures have affected SDA 
and service-provider behavior and how these 
effects have influenced the clients served, the 
services provided, and the costs of the ser- 
vices. This evaluation distinguished the ef- 
fects of several aspects of State policies, 
including (1) choices about procedures to ad- 
just standards, (2) policies for serving hard- 
to-serve clients, and (3) policies for awarding 
incentives for good performance and 
sanctioning for poor performance. 

When we found evidence that perfor- 
mance standards did appear to be influencing 
the clients, services, or costs, the study was 
designed to explore how and why those in- 
fluences occurred and whether the influences 
were intended (e.g., causing SDAs to berve 
hard-to-serve clients and to offer services 
that will make a difference in the long run) or 
unintended (for example, causing SDAs to 
serve fewer hard-to-serve clients or to offer 
less intensive services). 

Finally, the study was designed to 
generate suggestions for refining the perfor- 
mance-standards system so that any un- 
desirable impacts of performance standards 
can be prevented in Uie future. 

Overview of the Evaluation Design 

This evaluation was composed of two 
components. The first was a quantitative 
analysis of the effects of State and local 
policies using the sample of ail States and 
SDAs. This analysis provided objective, 
statistical evidence of the impacts of State 
performance-standards policies for the sys- 
tem as a whole. The quantitative evaluation 
examined whether differences in policies that 



SDAs face are associated with differences 
in the types of clients served, the services 
provided, or the programs' costs. The quan- 
titative analysis could not, however, provide 
evidence about the effects of Federal perfor- 
mance-standards policies, which do not vary 
among SDAs. Furthermore, why specific ef- 
fects occurred is difficult to determine fi:om a 
quantitative analysis. 

Thus, the second component of the 
evaluation was a qualitative evaluation, 
based on indepth case studies that included 
extensive onsite interviews and observations. 
The purpose of this component was to deter- 
mine the impact of Federal performance- 
standards policies and to determine why 
State, SDAs, and service providers have 
reacted to elements of the performance-stand- 
ards system. This qualitative evaluation was 
based on a sample of 30 SDAs and 87 ser- 
vice providers in 8 States. Although neces- 
sarily limited in number, these agencies were 
chosen to be as representative as possible of 
the JTPA system as a whole. 

Organization of the Report 

The next chapter presents the conceptual 
framework that guided the evaluation, includ- 
ing the hypothesized relationships between 
performance-standards policies and clients, 
services, and costs. Chapter in describes the 
Federal and State performance-standards 
policies that were evaluated. 

Section B presents the quantitative 
evaluation of ^e effects of performance 
standards that was based on surveys of all 
States and SDAs- Chapter IV presents the 
design of the quantitative component. Chap- 
ter V presents the impacts of State perfor- 
mance-standards policies and SDA practices 
on the types of clients served in JTPA 
programs. Chapter VI examines the impact 
of policies and procedures on the types of ser- 
vices offered and the costs of those services. 

Section C contains the results of the 
qualitative analysis. Chapter VII describes 



Introduction and Goa«3 of the Evaluation 



the design of the qualitative component. Be- 
cause the response to performance standards 
varied, Chapter VIII introduces the qualita- 
tive results by describing several typologies 
of SDAs* reactions to performance standards. 
Chapter IX presents case study results on 
how performance standards affected the 
SDAs' goals about whom to serve in JTPA 
programs. Chapter X describes performance 
standards* influence on how SDAs designed 
their service mix and program intensity and 
the factors that influenced program costs. 
Chapter XI presents the impacts of perfor- . 



mance standards on SDA management prac- 
tices, including the division of respon- 
sibilities between SDAs and service 
providers, the choice of providers, and the 
contractual arrangements. The qualitative 
analysis concludes with a description of 
SDAs' opinions of various aspects of the per- 
formance-standards system, presented in 
Chapter Xn. 

Section D presents a summary and con- 
clusions of the quantitative and qualitative 
analyses in Chapter Xin and the implications 
for the performance-standards system in 
Chapter XIV. 
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This evaluation is based on a conceptual 
framework that indicates the relationship 
among the various aspects of the JTPA 
program and specifies hypotheses that will be 
tested This firamework indicates not only 
that performance standards may affect the 
types of clients served, the types and length 
of services provided, and the cost of JTPA 
Title n- A programs, it also specifies several 
other relationships. This broader view is 
necessary for several reasons. First, the per- 
formance-standards system has severe dif- 
ferent dimensions , each of which may have a 
different effect on JTPA outcomes. Second, 
numerous elements of Federal, State, and 
SDA policies are also likely to influence 
clients, services, and costs. Some of these ele- 
ments are mechanisms through which perfor- 
mance standards affect program outcomes. 
Other elements arc confounding factors that 
are not related to the performance-standards 
system, but nevertfidess influence clients, ser- 
vices, and cost, and thus must be controlled 
in the evaluation. The conceptual framework 
is summarized in Figure 1, 



Participant Flow 

The right column in Figure 1 represents 
the flow of individuals through the program. 
First, individuals decide to apply to the 
program, in part on the basis of their 
knowledge of the program and their own as- 
sessment of the likely benefits of the program 
relative to those of alternative actions avail- 
able to them. A subset of applicants are 
selected for enrollment, either by die 
program (or its service providers) or by the 
applicants themselves. The clients enrolled in 
the program then receive specific types of ser- 
vices for various lengths of time. As a result, 
participating clients experience certain out- 
comes, such as finding a job or earning a cer- 
tain wage rate, and the program incurs costs. 
Thus, this column contains the dependent 
variables of the study; the enrollment of 
client groups, the types and length of services 
provided, and the costs of various services. 

Program Design 

The middle colunm in Figure 1 describes 
the elements of each SDA*s JTPA program 
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PROGRAM DESIGN 



PARTICIPANT FLOW 



FEDERAL POUCtES 

P«Horminc#-St«ndafdt Policies 

• CholM ol mMtUTM 

• L«v«l of ttandardt 

• Chmc(«fitUct in model 

Other JTPA Polidet 

• 15%«dminMraliveKmtt 

• Support cost nmtt 

• Reporting requirements 



STATE POLICIES 

• Non*Federal measures and 
standards 

• Adjustnrtem methodology 

• Incentives, technical 
assistance, and sanctions 

• Reporting requirements 

• UrvkageswRh other programs 



PiC CHARACTERISTICS 

• Extent ot employer orientation 

• Degree ot involvement wXh 
design 

• Degree ol concern with 
perfomiance standards 
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SDA CHARACTERISTICS 

• Perceptions about perfonfnance 
standards 

• Management quality 

• Slzeotidlocatlon 



PROCESS AND CRITERIA TO 
SELECT aiENTS 

• Who conducts outreach 

• Who conducts Intake 

• Estabishment ot enrollment ct^h.^ 



PLANNED PROGRAM SERVICES 

• EstabHshmentolYEC 

• Criteria for choosing program 
services 

• Coordination wlh other programs 



SERVICE-PROVIDER ARRANGEMENTS 

• Choice of service providers 

• Choice of contract type 

• Choice of contract terms 



APPLICATION 
TO PROGRAM 



ENROUMENT 
OF CLIENTS 
IN PROGRAM 



RECEIPT OF 
SERVICES 
(Including 
length of stay) 
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PROGRAM COSTS 

(including 
expenditure rotes) 



PARTICIPANT 
OUTCOMES 



ENVIRONMENTAL CHARACTERISTICS 

• Population characteristics 

• Labor market characteristics 

• Geographic characteristics 
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design hypothesized to influence clients, ser- 
vices, and costs. Program-design elements 
are important mechanisms through which per- 
formance standards, as well as other con- 
founding factors, affect program outcomes. 

One important design element is the 
process and criteria used by an SDA to select 
clients for JTPA. SDAs have considerable 
discretion in choosing clients, although they 
are constrained by the legislated require- 
ments that 90% of the emoUees must be dis- 
advantaged and that 40% of the expenditures 
must be on youth programs. SDAs may estab- 
lish specific criteria for client selection. For 
example, they may establish "equity-of-ser- 
vice" goals to ensure enrollment of those 
most in need of services; other SDAs may 
have established testing to ensure enrolhnent 
of participants with specific skill levels. In 
addition, the process by which clients are 
selected can also influence the types of * 
clients served. Some SDAs conduct their 
own outreach and intake programs, which 
may result in greater SDA control over client 
selection. Other SDAs choose to have their 
service providers conduct outreach and in- 
take. As a result, service providers' own 
preferences and the incentives created by con- 
tractual arrangements may have a strong in- 
fluence on who is enrolled. 

Another imp/)rtant design element is the 
planned mix of program services. These 
planned services are also affected by JTPA 
legislation: the program has a greater em- 
phasis on training than was the case under 
the Comprehensive Employment and Train- 
ing Act (CETA), public service employment 
is not included in JTPA programs, and train- 
ing stipends are rarely available because of 
limitations on support costs. Generally, 
SDAs design a mix of services that includes 
classroom training in occupational skills, 
basic skills remediation, on-the-job training 
(OJT), job-search assistance, and a limited 
amount of work experience. 

SDAs can choose a mix of services to 



focus on the needs of local employers, on the 
needs of individuals applying to the program, 
or on some combination of these needs. Fur- 
tiiermore, in designing youth programs, 
SDAs can choose services aimed at out-of- 
school youth, for whom employment is the 
more appropriate outcome, or at in-school 
youth, for whom other outcomes are more 
appropriate, such as attainment of youth 
employment competencies (YECs). As part 
of the JTPA legislation, PICs are responsible 
for approving YEC systems that reflect local 
needs for entry-level employment skills. 
PICs may choose to develop youth competen- 
cies in three areas: pre-employment work 
maturity/skills, basic education, and job- 
specific skills. The SDAs design programs 
aimed at achieving the various competencies 
approved by their PICs. 

The planned mix of services can affect 
several aspects of the participant flow. Most 
directiy, the planned program mix affects the 
types of services actually received by clients 
and the costs of those services. The planned 
program mix may also influence the types of 
mdividuals who apply to JTPA programs. 
For example, tiiose v. ho have no alternative 
oources of income (for example, neither wel- 
fare nor family support) may be less likely to 
apply to JTPA programs than they were to 
CETA because they have an immediate need 
for employment income; in-school youth are 
less likely to apply to SDAs that offer only 
employment-oriented programs to youth. 
Program mix is also likely to affect the types 
of clients enrolled in JTPA programs. If an 
SDA chooses employer-oriented programs, 
for example, it may be more likely to enroll 
job-ready applicants, particularly in OJT 
programs for which employers may make the 
decision of whom to enroll. 

The third element of program design in- 
volves the arrangements for the provision of 
serxices, including the choice of specific ser- 
vice providers, the types of contracts used, 
and the terms of those contracts. The major 
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categories of service providers include public 
schools, community-based organizations 
(CBOs), other nonprofit organizations, and 
profit-making organizations. The choice of 
service providers may affect ail outcomes of 
interest to this evaluation. Service providers 
can have a strong influence on who is en- 
rolled, particularly in SDAs in which 
providers conduct intake activities. CBOs, 
for example, are more likely to see their goal 
as serving the disadvantaged, particularly the 
groups they represent, while profit-making 
organizations may see their goal as finding 
the best workers for employers. Because of 
these differences in orientation, the type of 
service provider may also affect the length 
of services provided and, therefore, the cost 
of services. 

The type of contract may also have a sub- 
stantial impact on clients, services, and costs. 
Performance-based contracts are widely used 
in JTPA programs. Under these contracts, 
providers are paid a fixed amount for each 
participant that is placed in a job after ter- 
mination. Some SDAs make intermediate 
payments for participants that achieve 
specified outcomes, such as completion of 
training, but the provider does not receive 
full payment unless the participant is placed. 
Furthermore, the provider gets paid the same 
amount for each individual it places, regard- 
less of how long the individual participated 
in the program or how many training resour- 
ces were devoted to the individual. 

The use of performance-based contracts 
may influence the types of individuals en- 
rolled in the program. If the service providers 
conduct intake, they may have an incentive 
to choose the most job-ready individuals be- 
cause they will need to expend fewer resour- 
ces and have a higher probability of 
receiving full payment by placing such in- 
dividuals. The use of performance-based con- 
tracts may also effect the types of services 
provided. For example, performance-based 
contracts may reduce the length of stay in the 



program because the provider has an incen- 
tive to place individuals as soon as they are 
employable, rather than provide longer term 
services that might lead to higher quality 
placements. 

The terms of performance-based con- 
tracts also affect the incentives of service 
providers. Contracts may vary in the type of 
placement outcome for which full payment is 
received (e.g., 30-day retention, initial place- 
ment), the fraction of payment that is 
reserved for termination outcomes, and the 
wage rates required Some SDAs pay an addi- 
tional lump sum if a specific placement rate 
is attained. 

These contract terms may influence 
clients, services, and costs. For example, 
higher wage rate goals may increase incen- 
tives to enroll more employable clients but 
may also increase incentives to provide 
longer training. Reserving a large fraction of 
payment for placement and paying an addi- 
tional amount for meeting a placement goal 
may intensify the incentives to enroll the 
most job-ready individuals and to provide the 
minimum training to achieve the outcome. 

Influences on Program Design 

The left column in Figure 1 indicates the 
factors that may influence JTPA program 
design. This column contains the major inde- 
pendent variables of this evaluation: Federal 
and State aspects of the performance-stand- 
ards system. The column also contains con- 
founding factors that must be controlled for 
in the evaluation, including other Federal and 
Sta):e policies, the characteristics of local 
agencies, including the PIC and the local 
elected officials involved in the JTPA 
program, and characteristics of the local 
environment. 

Federal Policies 

The performance-standards system is a 
key element of Federal policy hypothesized 
to influence the design of JTPA programs. 
Three distinct aspects of Federal policy 
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should be distinguished: the choice of perfor- 
mance measures, the numerical level at 
which the standards are set, and the choice of 
variables included in the optional adjustment 
model. 

The Federal performance-standards sys- 
tem is intended to be neutral with respect to 
client mix. This goal is to be accomplished 
through the use of regression-based adjust- 
ment models, which adjust SDA performance 
standards according to the average national 
experience in serving various types of clients 
in various economic conditions. Thus, the in- 
tended effect of these models is that SDAs 
are "held harmless" for their decisions about 
whom to serve for those characteristics in- 
cluded in the model. 

Although the intent is to make the stand- 
ards neutral with respect to client mix, the 
Federal standards n:ay have an unintended 
impact on client selection critma for several 
reasons. First, the adjustment models account 
for important measurable characteristics, 
such as race, sex, education, and receipt of 
welfare, but within these categories of 
clients, SDAs may have an incentive to 
choose the most job-ready candidates. For ex- 
ample, among school dropouts, SDAs may 
choose those who are highly motivated to 
seek employment, who have the best ap- 
pearance, or v/ho do not have other barriers 
to employment, such as a criminal record 
Furthermore, the adjustment models, which 
represent average national experience, may 
not appropriately adjust for client characteris- 
tics for certain types of SDAs. 

Second, the specific mix of standards 
may influence client-selection criteria. For ex- 
ample, the youth performance standards in- 
clude both an entered-employment-rate 
standard and a positive-termination-rate 
standard, reflecting the fact that JTPA youth 
programs have multiple goals. This combina- 
tion may affect client mix, however, because 
SDAs can more easily meet their entered- 
employment-rate standard by enrolling out- 



of-school youth, while they can more easily 
meet their positive-termination-rate standard 
by enrolling in-school youth because "com- 
pleted major level of education" is counted as 
a positive termination. As a result, possibly 
only SDAs that enroll a substantial number 
of both types of youth can meet all of their • 
youth standards. 

Third, the level of standards may affect 
the types of clients enrolled If the standards 
are set high so that SDAs cannot meet their 
standards without substantially altering their 
behavior, then they may be particularly likely 
to select clients who appear to be most job 
ready on the basis of characteristics not in- 
cluded in the model. To avoid this problem, 
the level of standards was set in PY 86 to 
reflect a minimally acceptable level of perfor- 
mance (that is, the 25th percentile of ex- 
pected performance) rather than to reflect an 
average level of expected performance. None- 
theless, the question remains whether this 
level of performance is still too high to 
eliminate the incentive to enroll the most job- 
ready individuals. 

Performance standards are also hypoth- 
esized to affect planned program mix. Perfor- 
mance standards are intended to guide SDAs 
to choose adult programs that are employ- 
ment oriented (by means of the entered- 
employment-rate standard), that train clients 
for high-quality jobs (by means of the wage- 
at-placement standard), and that are cost-ef- 
fective (by means of the cost-per-entered- 
employment standard). With respect to youth 
programs, performance standards are in- 
tended to allow SDAs to provide an ap- 
propriate combination of employment-related 
programs (by means of the entered-employ- 
ment-rate standard) or other-outcome- 
oriented programs (by means of the 
positive-tamination-rate standard). In either 
case, the programs are intended to be cost-ef- 
fective (by means of the cost-per-positive-ter- 
mination standard). 

In addition to these intended effects. 
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however, performance standards may have 
unintended impacts on planned program mix. 
Because the employment measure is termina- 
tion based, SDAs may be induced to provide 
programs with immediate employment as the 
goal, such as job-search assistance. SDAs 
may be reluctant to provide programs that 
retrain clients for new occupations because 
terminees may take longer to find jobs in 
new occupations. The level of standards may 
also affect program mix if the cost-per- 
entered-employment standard is so low that 
more intensive interventions are not feasible. 
^ Performance standards may also affect 
service-provider arrangements. The intent of 
the performance-standards system is to 
provide incentives for SDAs to choose effec- 
tive service providers and to use cost-mini- 
mizing contracting arrangements to achieve 
a given level of participant performance. 
However, performance standards also may 
have made SDAs more conservative and 
averse to risks when making service-provider 
decisions. For example, PICs may be more 
likely to fund traditional service providers 
and less willing to fund innovative programs 
because they are less sure that performance 
of new providers would be high enough to 
allow the SDA to meet its standards. 

In addition to the Federal performance- 
standards system, other Federal JTPA 
policies are likely to have important influen- 
ces on program design. In particular, the 
JTPA legislation restricts the administrative 
costs to 15% and the total of support costs 
and administrative costs to 30% of expendi- 
tures. The administrative limit is likely to 
create a strong incentive for SDAs to use per- 
formance-based contracts because the total 
costs of such contracts can be counted as 
training costs. 

The limit on support costs and work ex- 
perience can also affect program design. 
This limitation most directly affects the types 
of services offered, limiting work-experience 
programs and support services offered to par- 



ticipants. Support cost limitations may also 
limit the clients' willingness to participate in 
long-term classroom training without any in- 
come support. These restrictions may also in- 
fluence service-provider arrangements, 
particularly insofar as support services can 
be included under a performance-based con- 
tract and thus can be counted as training 
costs. 

Federal JTPA reporting requirements 
may also affect clients, services, and costs. 
The JTPA Annual Status Report (JASR) re- 
quires SDAs to report the number of adult, 
welfare, and youth terminees that have 
various characteristics, as well as the number 
achieving performance-standard outcomes. 
The presence of a client characteristic on the 
JASR, regardless of whether the characteris- 
tic is included in the adjustment model, has 
been suggested to encourage SDAs to in- 
crease service to that group. The fact that 
type of service is not included in the JASR 
may remforce SDAs' discretion to design ser- 
vices appropriate to local needs. Further- 
more, if Federal reporting requirements 
impose a record-keeping burden on SDAs, 
they may increase the costs of the program. 

Furthermore, in PY 86, DOL mitiated 
post-program data collection requirements 
that will support post-program standards 
beginning in PY 88. This data collection ef- 
fort can be funded from State 6% funds for 
the first 2 years, but unless the legislation is 
amended further, it will be funded from the 
State administrative funds or SDAs' alloca- 
tions in the future. It will be important, there- 
fore, to assess the impact of these additional 
data collection costs on clients, services, and 
costs. 

State Policies 

State policies can also have an important 
influence on JTPA program design. State 
policies explicitly related to performance 
standards include the addition of State stand- 
ards to the Federal standards, the choice of 
whether to use the Federal adjustment model. 



Concq)tual Framework of the Evaluation 



the allowance of adjustments beyond the 
model, and the policies for incentives, techni- 
cal assistance, and sanctions. 

Several States have adopted additional 
performance standards, including post- 
program standards and "equity-of-service" 
standards, that specify groups to be served in 
proportion to their incidence in the eligible 
population. These additional standards can 
potentially affect all the elements of program 
design. For example, equity-of-service stand- 
ards may increase service to more disad- 
vantaged groups, whereas post-program 
standards may create incentives to enroll 
more job-ready applicants. The additional 
standards may influence the types of services 
offered. For example, post-program stand- 
ards may increase length of stay because 
SDAs may need to provide more intensive 
services. Additional standards may also in- 
fluence the choice of service providers. For 
example, equity-of-service standards may in- 
crease the use of CBOs as service providers 
because of their commitment to serving more 
disadvantaged individuals. 

The procedures whereby States adjust 
SDAs* standards are also likely to affect 
program design. The Federal adjustment 
model is designed to hold SDAs harmless for 
their choice of clients to serve. States that do 
not use this model or a similar adjustment 
procedure are likely to increase the incentives 
for SDAs to serve the more job-ready can- 
didates. In addition. States may also make ad- 
justments beyond the model, most commonly 
for serving the hard to serve or for providing 
longer term program services. 

States* policies for distributing 6% incen- 
tive payments may also affect JTPA design 
decisions. There are numerous dimensions to 
these incentive policies. For example, these 
policies may emphasize high performance 
by requiring that many standards be met to 
qualify for incentives, by providing little pay- 
ment for meeting and more payment based 
on the extent to which standards are ex- 



ceeded, by having SDAs compete for funds, 
or by giving bonuses for extraordinary perfor- 
mance. These policies may strengthen any in- 
centives to enroll the less hard to serve. On 
the other hand. States may use incentive pay- 
ments to encourage service to the hard to 
serve by requiring that 6% funds be used for 
the hard to serve, by setting aside a pool of 
6% funds that SDAs bid on for serving the 
hard to serve, or by exempting 6% funds 
fipom performance standards if they are used 
for the hard to serve. 

In addition to State performance-stand- 
ards policies, otiier State policies may affect 
JTPA design. In several States, for example, 
the JTPA program has been linked to work- 
welfare programs, designed to increase the 
employability of welfare recipients. As a 
result, SDAs in tiiose States may face strong 
nressure to enroll more welfare recipients and 
to focus program services on the needs of 
these individuals, particularly for basic skills 
training. 

States may also affect program design by 
encouraging coordination between JTPA and 
other State programs, such as educational 
programs or welfare programs. These 
linkages are likely to be particularly influen- 
tial if SDAs can combine JTPA funds with 
funds from other programs. States can also 
provide statewide management information 
systems that allow SDAs to monitor their per- 
formance; this ability is hypothesized to ac- 
centuate the incentives created by the 
jverformance-standards system. 

Local Agencies 

Local factors may also influence JTPA 
program design. A key element tiiat distin- 
guishes JTPA programs from CETA is the 
PICs* involvement in the design and ad- 
ministration of JTPA programs in each local 
area. The PICs* role is intended to assist 
JTPA programs in better serving the needs of 
employers and to guide the program away 
from a purely social service orientation. In 
many SDAs, PICs make important decisions 



13 



Introduction 



about all ?.spects of program design, includ- 
ing choices about clients, services, service 
providers, and types of contracting arrange- 
ments. 

The extent of PIC involvement varies 
greatly. Some PICs are the administrative en- 
tity of the local program and, therefore, may 
exert more control over the program. Further- 
more, the influence that a PIC exerts on 
program design depends on the PICs* orienta- 
tion toward serving local employers* needs 
compared with ser/ing the needs of the disad- 
vantaged in the area. The PICs may also ac- 
centuate the effects of performance standards 
if their members see meeting those standards 
and recaving incentive awards as a very im- 
portant goal. 

The other member of the JTPA partner- 
ship is a local elected official, usually a coun- 
ty or city leader. As with the PICs, the 
mfluence of local elected officials depends 
on the extent of their involvonent in the 
JTPA program, their orientation toward the 
needs of employers and participants, and the 
importance that they place on performance 
standards. 

Finally, characteristics of the SDA and its 
staff are likely to influence program design. 
First, SDAs that were prime sponsors under 
CETA may be less responsive to the new in- 
centives and directions of the JTPA program. 
As a result, these SDAs may be less likely to 
respond to performance standards in choos- 
ing client selection criteria, service mix, and 
service providers. 

Second, the size of the SDA can affect 
how much control the SDA retains over the 
design and operation of program elements 
and can also influence the size of the incen- 
tive awards the SDA can receive for good 
performance. 

The third SDA characteristic that may in- 
fluence JTPA design is the staff *s percep- 
tions about the meaning and requirements of 
performance standards and the adjustment 
model. In talking with JTPA administrators 



in training conferences, we have found two 
types of misconceptions about the model. 
First, some administrators believe that SDAs 
should serve client groups that have positive 
coefficients in the model because such coeffi- 
cients mdicate that JTPA programs are more 
effective for those client groups. Second, 
some administrators believe that SDAs 
should serve client groups with negative coef- 
ficients in the model because they believe 
that doing so will make their standards easier 
to meet. Several public interest groups have 
conducted extensive training in performance- 
standard issues to reduce misperceptions. An 
important element of this evaluation, there- 
fore, is to assess the need for more technical 
assistance in performance-standards issues. 

A fourth SDA characteristic that is likely 
to influence program design is the quality of 
management SDAs that are well managed 
are more likely to design and implement the 
type of program that is most appropriate for 
local needs and meet their performance stand- 
ards as well. In contrast, poorly managed 
SDAs may find it difficult to meet perfor- 
mance standards without substantially alter- 
ing the design of the program, perhaps by 
choosing the most job-ready applicants and 
providing short-term, "quick-fix** training. 
The quality of management may therefore be 
an important characteristic that interacts with 
the effects of performance standards. 

Local Environment 

The final set of factors that can influence 
the design of JTPA programs is the charac- 
teristics of the local environment. The charac- 
teristics of the eligible population and of the 
local labor market are likely to affect the 
types of individuals who apply to the 
program, the criteria used to select clients, 
and the typss of programs deemed ap- 
propriate for local needs. Furthermore, 
geographic characteristics (for example, rural 
locations) are likely to influence the ability 
of SDAs to provide specialized services and 
to limit the types of service providers avail- 
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able. 

Unlike the other factors hypothesized to 
influence program design, local environmen- 
tal factors are likely to affect clients, services, 
and costs directly , regardless of their in- 
fluence on program design. The characteris- 
tics of the eligible population will clearly 
influence who applies to and enrolls in the 



program. Local areas with higher unemploy- 
ment may have more job-ready applicants 
who may also stay longer in the program. 
Program costs may be higher in rural areas 
because of transportation problems. Thus, 
local environmental factors are potential con- 
founding variables that must be controlled 
for in the evaluation. 
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Introduction 

This chapter presents an overview of the 
Federal and State perfonnance-standards 
policies evaluated in this study. We describe 
the legislative requirements, die policies in ef- 
fect for PY 86, and the intention of these 
policies, as gathered from interviews with 
Federal and State staff. We found a great deal 
of variation in State policies and, therefore, 
present the number of States with different 
types of policies. 

Federal Policies 

The Federal role in the performance- 
standards system is to choose performance- 
standards measures, to set the level of 
performance standards, and to establish 
parameters within which standards can be ad- 
justed, including the provision of optional 
models to adjust SDAs' perfomiance stand- 
ards. The following section describes the 
legislative requirements for these three areas, 
the Federal policies for P Y 86, and how these 



policies have changed over time. 

Choice of Measures 

The JTPA legislation calls for the 
Secretary of Labor to prescribe performance 
standards to measure the increase in enq)loy- 
ment and earnings and the reduction in wel- 
fare dependency resulting from participation 
in the prognua For adults, the legislation 
states that the measures ^ Wy include (A) 
placement in unsubsidized employment, (B) 
retention in unsubsidized employment, (C) 
increase in earnings, including hourly wages, 
and (D) reduction in the number of in- 
dividuals and families receiving cash welfare 
payments and amounts of such payments/' 
The legislation also states that the Secretary 
''shall prescribe performance standards relat- 
ing gross program expenditures to various 
performance measures.'' 

From the beginning of JTPA through 
PY 87, there have been four adult perfor- 
mance measures: the entered-employment 
rate for all adults, the entered-employment 
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rate for adults receiving welfare at enroll- 
ment, the average wage rate at placement, 
and the cost per entered employment. 

DOL chose the level of wage rates rather 
than the change in wage rates because most 
JTPA participants were not working at enroll- 
ment and, in fact, nearly half had been un- 
employed 15 weeks or more before enroll- 
ment. Thus, DOL concluded it was too dif- 
ficult to measure accurately a pre-enrollment 
wage rate for JTPA participants. DOL inves- 
tigated the feasibility of an explicit welfare- 
reduction measure, but State variability in 
both AFDC and general assistance eligibility 
rules and benefit levels made such a measure 
incomparable across States. Because the 
reduction in welfare was caused by increased 
employment, the percentage of welfare 
recipients that entered employment was 
chosen as a measure of welfare reduction. 

Although DOL initially requested collec- 
tion of data on participants' employment 
status 3 months after termination, such data 
collection was turned down by the Office of 
Management and Budget (0MB). Beginning 
in PY 86, however. States and SDAs began 
to collect this information. In PY 88, DOL 
added four post-program measures: the per- 
centage of all adults who are employed 13 
weeks after t^nunation, the percentage of 
adults receiving welfare at enrollment who 
are employed 13 weeks after termination, the 
average number of weeks worked by all 
adults m the 13 weeks after termination, and 
the average weekly earnings of adults 
employed in the 13th week. 

The legislation also addresses youth per- 
formance measures and states that '*the 
Secretary shall also designate factors for 
evaluating the performance of youth 
programs, which ... shall be (A) attainment of 
recognized employment competencies recog- 
nized by the private industry council, (B) 
elementary, secondary, and postsecondary 
school completion, and (C) enrollment in 
other training programs or apprenticeships. 



or enlistment in the Armed Forces" in addi- 
tion to appropriate measures listed for adults. 

Again since the beginning of JTPA, DOL 
has prescribed three youth performance 
measures: entered-employment rate, posi- 
tive-termination rate, and cost per positive 
termmation, A positive termination is 
defined as entering employment, the three 
factors listed above, returning to full-time 
school, and for 14- to 15-year-olds only, com- 
pleting program, objectives. 

DOL has become increasingly concerned 
about the problems of youth illiteracy and 
lack of other basic educational skills. Thus, 
beginning in PY 88, DOL added a fourth 
youth performance measure, employability 
enhancement. This measure is similar to posi- 
tive termination except that entered employ- 
ment is not counted as an employability 
enhancement. The intention of this additional 
measure is to encourage SDAs to address the 
barriers that youth face in attaining quality 
employment, in addition to finding them im- 
mediate employment. Furthermore, begin- 
ning in PY 89, SDAs can claim attainment of 
youth employment competencies only if par- 
ticipants attain competency in at least two of 
three areas: pre-employment/work maturity 
skills, basic educational skills, and occupa- 
tion-specific skills. The intention of this 
policy is to encourage SDAs to address the 
multiple barriers to employment that face 
many youth. 

Because of these changes for PY 88, 
there will be 12 Federal performance stand- 
ards instead of the 7 used previously. To 
avoid burdening the system, DOL has 
directed States to choose eight of these 
standards. 

Numerical Level of Standards 
The Secretary also sets the numerical 
level of the performance standards. In fact, 
there are two separate choices in levels: the 
levels of the national standards and the levels 
fix)m which SDAs* standards are adjusted in 
the optional adjustment models. The national 
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standards represent the performance goals for 
the JTPA system as a whole. The levels used 
in the adjustment models, called departure 
points, represent the average model-adjusted 
standards. Because the departure points and 
the national standards serve different func- 
tions, their values need not be identical. 

Originally, the performance levels for 
JTPA programs were based on the average 
level of performance in CETA, under the as- 
sumption that JTPA programs would perform 
at least that well. DOL decided, however, not 
to set performance standards at the average 
performance for the JTPA program. Doing so 
would mean that half the SDAs would be ex- 
pected to fail their standards, and below 
average perfonnance was not necessarily 
failure. Furthermore, it was feared that set- 
ting performance standards at a high level 
would place too much emphasis on perfor- 
mance at the expense of other program goals. 

Beginning in PY 86, therefore, JTPA 
departure points were set at the 25th percen- 
tile of past performance relative to adjusted 
standards, a level that represented minimally 
acceptable performance. The national stand- 
ards were also set at the 23th percentile for 
five of the seven measures. The national 
standard for the wage rate was set at a higher 
level to indicate tiiat die goal of the program 
is to train participants for jobs that pay well, 
and the national standard for the youth cost 
standard was set at a more lenient level to em- 
phasize that cost considerations should not 
prevent providing needed services to youth. 
The departure points and national standards 
for PY 86 are presented in Table 1. 

For PY 88, DOL is continuing th». policy 
of setting the national performance standards 
and departure points at the 25th percentile 
with two major exceptions. In response to 
concerns that the mandated cost standards 
were reducing the intensity of JTPA services, 
DOL set botii die adult and youtii cost stand- 
ards at levels considerably more lenient (at 
approximately die 5di to 1st percentiles. 



respectively) to provide a signal to die sys- 
tem that the cost standard should not inhibit 
the local decision about the intensity of ser- 
vices. To emphasize the importance of obtain- 
ing quality jobs, the wage-at-placement 
standard was set at a higher level 

Adjustment Parameters and 
Optional Adjustment Models 
The JTPA legislation states diat "Each 
Governor may prescribe, within parameters 
established by the Secretary, variations in die 
standards ... based upon specific economic, 
geographic, and demographic factors in die 
State and in service delivery areas within die 
State, the characteristics of die population to 
be served, and die type of services to be 
provided." To facilitate this process, the 
Secretary has issued parameters for adjust- 
ment procedures and optional adjustment 
models for the standards that meet these 
parameters. 

DOL*s adjustment parameters were 
qualitative rather than quantitative to give 
Governors considerable flexibility in setting 
SDAs* standards. The qualitative parameters 
require that procedures for adjusting perfor- 
mance standards must be 

• Responsive to die intent of die JTPA 
legislation, 

Consistenfly applied among SDAs, 

Objective and equitable throughout 
the State, and 

In conformance with widely accepted 
statistical criteria. 
Source data must be 

Of public use quality and 

Available upon request 
Results must be 

Documented clearly and 

Reproducible. 
Adjustment factors must be limited to 

Economic factors. 

Labor market conditions, 
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Table 1 

Level of National Standards and Departure Points for PY 86 



Departure Point 



National Standard 



Adults 

Entered-employment rate 

Wage at placement 

Cost per entCTed employment 

Welfare 

Entered-employment rate 

Youth 

Entered-employment rate 
Positive-termination rate 
Cost per positive termination 



62.4% 
$4.64 
$4,374 



51.3% 



43.3% 
74.8% 
$3,711 



62% 
$4.91 
$4,374 



51% 



43% 
75% 
$4,900 
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• Characteristics of the population to be 
served, 

• Geographic factors, and 

• Types of services to be provided. 

DOL's optional adjustment models con- 
form to these parameters. The optional adjust- 
ment models are based on an analysis of the 
relationship between variation in the perfor- 
mance of SDAs and variation in the types of 
clients served and In local economic condi- 
tions. The adjustment models indicate how 
much performance is affected by these fac- 
tors and adjusts the performance standards 

of the SDA accordingly. 

The models are based on JTPA data on 
performance and the characteristics of clients 
served, as reported in the JASR. Thus, the 
client groups for which adjustments can be 
made are restricted to those groups reported 
on this form. DOL has gradually added new 
client groups, but 0MB is concerned about 
reporting burden and in the past has turned 
down many requested items. Beginning in 
PY 88, however, SDAs will report the num- 
ber of tenninees with below 7th grade read- 
ing level and those receiving AFDC for 2 or 
more years. Client characteristics included in 
die PY 86 adjustment models are presented 
in Table 2. 

Local economic conditions are calculated 
for each SDA using data from Census or the 
Bureau of Labor Statistics (BLS). The 
economic conditions included in the PY 86 
models are also included in Table 2. 

For several reasons, the DOL adjustment 
models do not adjust for differences in the 
types of services offered. First, it was felt that 
die provision of appropriate services was an 
important management decision for which 
SDAs should not be held harndess in the ad- 
justment process. Second, there is no uniform 
definition of types of services, and there was 
a concern that imposing Federal uniformity 
might reduce local flexibility in program 
design. Third, there is no information on the 



JASR to make such adjustments. 

Because of DOL 's increased concern 
about the intensity of services offered in 
JTPA programs, consideration was given to 
adding some adjustment for the provision of 
basic skills training, particularly to the cost 
standards. DOL did not request the addition 
of the percentage of terminees receiving 
basic skills training for the most recent JASR 
revision, however, in part because no data 
were available to document that SDAs that 
provide this type of training perform less 
well on the cost standards. The data set 
gathered for this study is the first that con- 
tains data on both costs and program services 
to examine these relationships. 

DOL also issues a tolerance range for 
each performance measure that represents tlie 
average amount of imprecision in the models. 
DOL considers any additional adjustments 
made within the upper and lower bounds of 
the tolerance range to meet the Secretary's 
parameters. States have used the tolerance 
range in a variety of ways, as described in the 
next section. 

In addition to the adjustment models, 
DOL also has provided a technical assistance 
guide that describes valid procedures for 
making adjustments for client characteristics 
not included m the model, for special 
programs, and/or for SDAs facing extreme 
circumstances. DOL has encouraged States 
to establish policies for "adjustments beyond 
the model." Furthermore, several training ses- 
sions have also presented these adjustment 
procedures to State and SDA staff. 

State Performance-Standards 
Policies 

The JTPA legislation and regulations 
specify important roles for States in the per- 
formance-standards system. States may 
specify additional performance measures, set 
standards for SDAs, establish policies to 
award incentive funds to SDAs with good 
performance, and sanction SDAs with poor 
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Table 2 

Local Characteristics Included in PY 86 Adjustment Models 





Adult Models 


Youth Models 


Terminee 
characteristics 


% Female 


% Female 




% 55 years & over 


% 14-15 years 




% Black 


% 18 -21 years 




% Hispanic 


% Blark 




% Alaskan Native/Ajnerican 
Indian 

% Asian/Pacific Islander 


% Hispanic 

% Alaskan Native/American 
Indian 




% Dropout 


% Asian/Pacific Islander 




% Handicapped 


% Dropout 




% UC claimant 


% Student 




% Welfare recipient at entry 


% High school graduate 




Average weeks participated 


% Post high school attendee 
% Handicapped 
% Offender 

% Welfare recipient at entry 
Average weeks participated 


Local economic 
conditions 


Average wage for area ($1000*s) 


Average wage for area ($1000's) 




Unemployment rate 


Unemployment rate 




% Families below poverty level 






Population density 
(lOOOVsq.mi.) 





Note: Some characteristics are not included in all models. 
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performance. 

In addition, many States have established 
policies for serving hard-to-serve groups, and 
some of these policies involve performance- 
standards mechanisms. In this section, we 
present the variation in State policies that we 
found from our policy review and discuss the 
intentions of these policies as reported in our 
site visits . the eight States in the qualitative 
study. 

State Performance Measures 

Eighteen States adopted additional perfor- 
mance standards for PY 86.* Table 3 
presents the types of State standards used 
Seven States had established post-program 
standards for P Y 86, 2 years before post- 
program standards were included in the 
Federal measures. Six of the States used a 
measure of employment at follow-up, al- 
though earnings at follow-up, weeks worked, 
and welfare follow-up employment rates 
were also adopted. All of these measures are 
included as Federal measures for PY 88. 

Nine States adopted performance stand- 
ards intended to encourage service to hard-to- 
serve groups, including welfare recipients, 
dropouts, minorities, women, handicapped, 
and older workers. In all cases, these stand- 
ards were for a required ic^vel of service to 
these groups, although some States had re- 
quired placement rates for these hard-to- 
serve groups as well. 

Two States added adult termination- 
based measures, the gain in wage rates and 
an entered-employment rate for training ac- 
tivities. Two other States extended the 
Federal adult termination-based measures to 
youth; one required a youth wage-rate stand- 
ard and the other a cost-per-entered-employ- 
ment standard. One State used JTPA 
performance standards to further economic 
development activities, and another estab- 
lished a standard for coordination with other 



social service agencies. 

Two States established performance 
standards based on the percentage of the 
SDAs' allocation expended. As discussed 
below, several other States incorporated ex- 
penditure rate requirements into the calcula- 
tion of incentive payments as an alternative 
to establishing a standard for this require- 
ment. 

Among the eight States in our case study 
sample, four had adopted State standards (or 
policies that functioned as standards) for PY 
86 and another had added a State standard for 
at-risk youth for PY 87. One case study State 
took a very active stance in using the stand- 
ards to further its goals. To reduce reliance 
on direct placement services, this State 
adopted a separate standard for the placemcrt 
rate for training activities; to reduce the ten- 
sion in the youth standards, it set separate 
positive-termination-rate and entered-employ- 
ment-rate standards for employment-oriented 
and remediation-oriented programs; to en- 
courage service to specific groups, it awarded 
30% of its incentive payments based on ser- 
vice to dropouts and minorities. 

Another State established a standard for 
linkages with other agencies, a direct out- 
growth of the Govemor*s concern to reduce 
duplication of services. In PY 88, tiiis State 
will also adopt a standard based on service 
to hard-to-serve welfare recipients, again in 
direct response to the Govemor*s priorities. 
Another State in the case study also awarded 
incentives based on service to hard-to-serve 
groups. 

The fourth State established a standard re- 
lated to adult and youth expenditure rates and 
enrollment rates to focus effort on com- 
pliance more than performance. A fifth State 
also incorporated expenditure rates into the 
calculation of incentive awards but did not 
establish a separate standard. 



*In some cases these goals were not explicitly called standards, but a proportion of incentive funds was 
awarded based on whether SDAs met goals and thus the goals functioned as standards. 



Table 3 

Types of Additional State Performance Standards 



Number of States 



Post-program standards 7 

Adult post-program employment 6 

Adult post-program earnings 1 

Weeks worked 1 

Welfare post-program employment 1 

Standards for hard to serve 9 

Welfare 3 

Dropouts 3 

Minorities 3 

Women 3 

Handicapped 3 

Older workers 3 

Hard to serve in general 3 

Youth standards 2 



Cost per entered 
Wage at placement 



Adult termination standards 
Wage gain 

Entered employment for training 
activities 



Economic development 1 
Linkages with other agencies 1 
Expenditure rates 2 
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Increasingly, States are using the perfor- 
mance-standards system to further their own 
goals for the JTPA system. Generally, we 
found that the State priorities were well ac- 
cepted by the SDAs; in fact. State standards 
had often been developed in consultation 
with SDA representatives. 

State Adjustment Policies 

The DOL adjustment models were widely 
used in PY 86. Forty-two States used the 
models for all of the Federal performance 
standards.* Of the eight States that did not 
use the DOL models for any standard, three 
were single SDA States.** 

Of the States that used DOL models, 26 
had a stated policy to allow for additional ad- 
justments beyond the model, although only 
15 of the States explicitly described the pro- 
cedures that SDAs should follow to request 
an adjustment. Despite the existence of State 
policies for adjustments, only 15% of all 
SDAs applied for adjustments in PY 86; of 
those requests, approximately 75% were 
granted. These adjustments were for a variety 
of circumstances, including special pro- 
grams, economic conditions, and unexpected 
events. 

For PY 86, all eight case study States 
used the DOL adjustment models for all 
standards, although one State made major 
modifications to the model, as described 
below. Two States made the effort to develop 
their own alternative models using State- 
specific data. In one State, the alternative 
model explained only 2% more of the varia- 
tion in SDA performance than the national 
model, so the effort was abandoned. The 
other State plans to test its alternative model 
in PY 88 but not to base incentive awards on 
those calculations. This State's model differs 
from the national model principally in that it 



adjusts for the type of service activities. 

Some case study States adopted policies 
for additional adjustments to encourage 
SDAs to serve the clients and offer the ser- 
vices appropriate to local needs. 1 "owcver, 
SDAs appeared to be reluctant to request 
such adjustments, and the States found it dif- 
ficult to develop clear criteria for determining 
tlie amount of adjustment 

Adjustments were used in two other ways 
among the case study States. First, States ad- 
justed standards "after the fact" to forgive 
particular SDAs for failing to meet a par- 
ticular standard by a tiny amount. Second, at 
least two case study States initiated across- 
the-board adjustments to alter the impact of 
performance standards on all SDAs in the 
State. One State reduced the effective level 
of all standards using this method to mini- 
mize the impact of the standard on SDA be- 
havior, another State increased the effective 
level of several standards to "challenge" the 
SDAs with the standards. 

Finally, some States appeared to be reluc- 
tant to permit any adjustments to the stand- 
ards at all because of the fear of contam- 
inating the "objective" DOL model with 
subjective factors. 

Incentive Policies 

The JTPA legislation gives States the 
authority to distribute up to 6% of JTPA 
funds as incentive grants for good perfor- 
mance, stating that these funds "shall be used 
by the Governor to provide incentive grants 
for programs exceeding performance stand- 
ards, including incentives for serving hard-to- 
serve individuals." The legislation further 
states that incentive grants "shall be dis- 
tributed among service delivery areas within 
the State exceeding their performance stand- 
ards in an equitable proportion based on the 



*DOL provided two procedures to adjust the welfare entered-cmployment rate: an adjustment model and an ad- 
justment based on the State ratio of the welfare entered-cmployment rate to the entered-employment rate for all 
adults. We counted the use of either procedure as use of the DOL model. 

♦♦Several single-SDA States used the models to set standards for their substate areas or providers. 
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degree by which the service delivery areas 
exceed their performance standards." Thus, 
the legislation explicitly requires that incen- 
tive funds be based on exceeding, not just 
meeting, performance standards. The 6% 
funds not used for incentives are to be used 
for technical assistance to SDAs that do not 
c *fy for incentive grants. In addition, 
Sutes may set aside a portion of 6% funds 
up front for preventative technical assistance. 
Overall, States used an average of 14% of the 
6% funds for all types of technical assistance 
inPY 86. 

The States have developed incentive 
policies that vary widely on a number of 
dimensions. First, States vary in the criteria 
they have established for SDAs to qualify for 
any incentives. Thirty-two States consider an 
SDA to have exceeded its standard for incen- 
tive purposes if its performance is above the 
model-adjusted level (or the national stand- 
ard). However, DOL publishes a tolerance 
range around the standard tiiat indicates the 
amount of statistical imprecision in the 
models. Nine States require that SDAs per- 
form above the upper level of that tolerance 
range, and four require that SDAs perform 
above the lower level of the tolerance range 
to qualify for incentives.* 

States also specify the number of stand- 
ards that must be exceeded to qualify for in- 
centives. On average. States require that half 
the standards be exceeded, although 14 
States require only one standard and 7 re- 
quire that all Federal standards be exceeded 
for SDAs to receive incentive funds. Nine 
States also require core standards be ex- 
ceeded, most commonly the adult and wel- 
fare entered-employment rates. 

The formula for determining the amount 
of incentives also varies widely. Twenty-two 
States have established separate pools of in- 



centive funds for each standard. The other 28 
States combine the performance on all stand- 
ards into a summary measure. Most common- 
ly this is done by summing the percentage by 
which each standard is exceeded. This proce- 
dure gives more weight to standards with 
greater variation, specifically the cost 
measures. Thus, an unintended effect of this 
procedure is to base the incentive funds more 
heavily on how much SDAs exceed their 
cost standards. This point was mentioned by 
a number of SDA directors that we inter- 
viewed in the case studies. 

Besides this de facto weighting proce- 
dure, 20 States mtentionally weight perfor- 
mance on some standards more heavily, so 
that on average the adult measures are 
weighted more heavily than the youth 
measures, and the entered-employment rates 
are weighted more heavily than other out- 
comes. The combined effects of intentional 
and unintentional weights are presented in 
Table 4. 

Finally, States determine the formula that 
awards 6% funds based on the "degree by 
which service delivery areas exceed their 
standards." Eighteen States have taken a very 
literal approach to this requirement and 
award 6% based on a continuous function of 
the extent to which the standards are e/- 
ceeded Thus, in these States, SDAs that just 
marginally exceed their standards do no^ 
receive any incentive awards, and SDAs 
receive more incentives the more they ex- 
ceed their standards. (For example, several 
States calculate the total percentage exceeded 
across all SDAs that qualify for incentives 
and award to each SDA the fraction of the 
total that they achieved.) 

In contrast, other States have established 
several categories of exceeding standards, so 
that, for example, SDAs that exceed their 



♦Some States have a two-tier system whereby SDAs must perfonn above the lower level on a large number of 
standards and above the upper level for a smaller number. The discussion pertains to the most stringent level in 
these cases. 
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Table 4 

Mean and Range of Weights Placed on Federal Performance Standards 



Mean Minimum Maximiun 



Adult 

Enteied-employment rate 
Wage rate at placement 
Cost per entered employment 
Welfare entered-employment rate 

Youth 

Entered-employment rate 
Positive-termination rate 
Cost per positive termination 



13.1% 3.8% 25.0% 

11.3 3.7 35.9 
18.2 4.7 32.7 

14.4 7.6 26.3 

17.2 5.0 41.0 

9.9 2.0 16.5 

15.8 1.0 23.3 
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Standards by less than 5% receive a certain 
award, those that exceed by 5 to 10% receive 
more, and those that exceed over 10%.., 
receive the maximum award. Furthermore, in 
30 States, SDAs compete among themselves 
for incentives, so that the better one SDA 
does, the less incentives the other SDAs 
receive. 

To capture these different dimensions, 
we coded several elements of the incentive 
policy, including the percentage of incentive 
funds that an SDA would receive for just 
marginally exceeding its standards, whether 
there is a threshold beyond which SDAs 
would not receive additional incentives for 
further exceeding its standards, and whether 
SDAs receive a bonus beyond the regular 
formula for very high performance, for 
example, for being the highest m the State. 
The number of States with these different 
dimensions is presented in Table S. 

In addition to awarding performance. 
States also use 6% funds to further other 
State objectives. Fifteen States include expen- 
diture rate requirements in calculating the 
amount of incentives that SDAs receive, for 
example, by multiplying the amount of incen- 
tives an SDA would receive based on perfor- 
mance by the SDA's expenditure rate. 

States also use 6% ^nds to encourage 
service to hard-to-cerve groups. Seven States 
have set aside part of the 6% funds at the 
State level to be used to serve the hard to 
serve. Five States have incorporated service 
to the hard to serve into the 6% calculation, 
for example, by multiplying the award by the 
extent to which the SDA served speciflc 
groups in proportion to their incidence in the 
eligible population. Alternatively, three 
States require some of the incentive funds be 
used for the hard to serve, and eight States 
omit programs funded with incentive funds 
from performance standards if they are used 
for the hard to serve. 

The case study States seemed to have 
several different reasons for their choice of 



6% policies. First, many of the case study 
States wanted to "develop the best possible 
"technical" solution to the question of how 
to weight the different standards and how to 
come up with a summary performance 
measure to determine the share of 6% funds 
that SDAs would receive. In pursuing this ob- 
jective, technical performance-standards staff 
were assigned the job of reviewing the pos- 
sible alternatives and coming up with a staff 
reconmiendation. Many of these staff 
reviewed practices in neighboring States and 
combined elements of policies in several 
other States. Elaboration followed elabora- 
tion, and before long. State incentive policies 
became highly complex documents. 

For States treating the development of 
6% policy primarily as a technical question, 
State staff did not appear to have spent much 
time worrying about whether SDAs would be 
able to grasp the intricacies of the 6% policy 
or what effect this policy would have on 
SDA policies and practices. Despite the lack 
of specific intent behind many State 6% 
policies, both the qualitative and quantitative 
research suggest that the choice of 6% 
policies has a strong effect on SDA strategic 
choices* 

A smaller number of States viewed the 
6% incentive award policy primarily as a 
way to influence SDA behavior. At least 
three strategies were used. One State wanted 
all its SDAs to succeed on the standards and 
wanted to minimize the influence the stand- 
ards might have on SDA behavior, viewing 
performance standards as primarily a public 
relations vehicle. Tliis State awarded only 
one-third of the 6% funds based on perfor- 
mance (the other two-thirds were based on 
size and expenditure rates) and used the 
lower boundary of each tolerance range. 

Several States made a clear policy deci- 
sion to give the standards unequal weight in 
their incentive awards policy. These States 
were trying to influence SDA decisions 
about tradeoffs among standards. A final 



Table 5 

Number of States with Various 6% Policies 



Number of States 



Percent of incentive funds received 
for marginally exceeding standards: 



None 18 

1-25 5 

26-50 8 

51-75 8 

76-90 5 

91% or more 3 



Has cap on performance, beyond wliich 21 
no further incentives received 



Gives bonus for extraordinary 
performance 



SDAs compete for amount of 30 
incentives received 



Note: Four single-SDA States without a clear incentive policy 
are excluded from this table. 
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group of case study States actively discour- 
aged tradeoffs among the different standards 
by requiring that SDAs meet all seven stand- 
ards before qualifying for any incentive 
award. It is not clear, however, that the inten- 
tion of these States was to reduce the pos- 
sible strategic performance options available 
to the SDA. One Staters intention was mere- 
ly "to reward performance above the norm 
rather than minimally acceptable 
performance." 

Another key dimension on which State 
policies varied has to do with whether the 
State emphasized performance at the level of 
the standards or whether its awards policy en- 
couraged overperformance. As we describe 
in subsequent chapters, this is one of the 
variations that had strong effects on the 
SDAs in each State. There was litde evi- 
dence, however, that States considered the in- 
fluence this aspect of the 6% policy might 
have on SDAs when they made the choice to 
emphasize performance at the standards level 
or to offer significant rewards for overperfor- 
mance. States appeared more concerned that 
their procedures were technically correct or 
that they could identify exemplary 
performers. 

Sanction Policies 

The JTPA legislation also requires that 
States sanction SDAs for poor performance, 
stating "The Governor shall provide techni- 
cal assistance to programs v/hich do not meet 
performance criteria. If tiie failure to meet 
performance standards persists for a second 
year, the Governor shall impose a reorganiza- 
tion plan." 

States have been slower to develop sanc- 
tion policies, and as of PY 86, nine States did 
not have a specific policy. Of those tiiat did, 
21 defined failure to meet a standard as per- 
forming below die standard, and 16 States 



defined failure uS performing below the 
lower level of tiie tolerance range. On 
average, SDAs were considered to have 
failed their standards in 1 year if they missed 
any three standards, altiiough seven States 
defined a core set of standards tiiat must be 
met, most commonly including the adult and 
welfare entercd-employment rates. Twenty- 
two States defined failing 2 consecutive 
years as missing the same number of stand- 
ards and eight required tiiat tiie same stand- 
ards be missed for 2 years. 

Across tiie eight case study States, one 
rare area of similarity was in tiieir overall ap- 
proach to sanction policies: to hope it would 
never happen. There is an ovenicUng posture 
of not wanting to be heavy handed. 

Two of tiie States visited had not devel- 
oped policies because the SDAs had per- 
formed so well that they could not foresee 
tiie issue arising. In two additional States, tiie 
issue of sanctioning was nearly ignored. One 
of those had developed clear criteria for SDA 
eligibility for incentive funds, but when one 
SDA failed to meet those criteria 2 years in a 
row, they were not specifically sanctioned. 
The remaining States had spelled-out policies. 

Only 2 SDA^ in our sample of 30 had ex- 
perienced such difficulties witii performance 
tiiat tiiey met tiie Staters criteria for failure 
and might experience sanctioning if that per- 
formance were repeated in a second year. 
One of those SDAs reported that the State 
had been extremely helpful in designing and 
monitoring a corrective action plan, a learn- 
ing process tiiat tiie SDA found beneficial. 

It appears tiiat States have spent very lit- 
tie time worrying about sanctioning SDAs. 
Instead, tiiey have focused on developing 
monitoring procedures and corrective action 
plans in those instances when it appears that 
performance may be problematic. 
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Introduction 

The purpose of the quantitative evalua- 
tion is to provide objective statistical 
evidence about the impact of State and SDA 
policies and practices on clients, services, 
and costs. The quantitative evaluation was 
designed to determine the following relation- 
ships: 

• Overall impact of State performance- 
standards policies on clients, services, 
and costs; 

• Impact of other factors, including con- 
cerns and influence of PIC and local 
elected officials (LEO) and charac- 
teristics of local eligible population 
and economy, on clients, services, 
and costs; 

• Impact of SDA practices, including in- 
take procedures and service provider 
arrangements, on clients, services, 
and costs; and 



• Impact of State performance-stand- 
ards policies on SDA practices that 
are found to affect clients, services, 
and costs. 

This chapter describes the design of the 
quantitative analysis, including the strategy 
for collecting the data and the analysis 
strategy to determine these relationships. 

Data Collection Strategy 

The quantitative evaluation required data 
on program outcomes (clients, services, 
costs). State perfomancc-standards policies, 
SDA practices (enrollment criteria, types of 
service providers, contracting procedures), 
PIC and LEO characteristics, and local en- 
vironn^nt characteristics, particularly charac- 
teristics of tiie local population eligible for 
JTPA. Existing data were used to the extent 
possible, but new data were required for 
several important aspects of tiie JTPA sys- 
tem. Tile strategies for using existing data 
and collecting new data are drscribed below. 
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Use of Existing Data Sets 

The quantitative evaluation made use of 
three existing data sources: the JASR, the 
Census Bureau, and BLS. 

The JASR provided information on the 
characteristics of terminees, the length of par- 
ticipation, and the costs of the program. 
These data are reported separately for all 
adults, welfare adults (except costs), and 
youth. Each SDA reports these data to the 
States and then to DOL. 

The JASR data were used to measure key 
outcomes for the evaluation, including 
characteristics of clients served, the length of 
program participation, and costs. The JASR 
does not, however, contain any data on types 
of services received or on any of the inde- 
pendent variables required for the evaluation. 
Nonetiieless, SDA and State identifiers are 
provided for tiie JASR data so tiiat these data 
were linked to other data sources for the 
evaluation. 

The other sources of existing data used 
for this study were the Census Bureau and 
BLS. This study required data on the charac- 
teristics of tiie local eligible population, the 
local labor market, and geographic charac- 
teristics. 

Two sources from the 1980 census were 
used to develop measures of the characteris- 
tics of tiie eligible population: the Summary 
Tape Files and tiie Public-Use Microdata 
Samples. The Summary Tape Files contain 



tabulations of data from tfie 1980 census so 
that the constmction of characteristics is 
limited by tiie specific tabulation provided. 
The percentage of tiie local population witii 
income below poverty and the population 
density of tiie SDA were obtained from tiiis 
source. However, tiie characteristics of the 
local eligible population available on Sum- 
mary Tape Files were not detailed enough for 
this evaluation. 

To calculate the characteristics of tiie 
eligible population, we used tiie Public-Use 
Microdata 5% Sample Tapes of individuals. 
From tills detailed individual-level file, we 
calculated the percentage of the eligible 
population with each of the characteristics 
listed in tiie JASR. We calculated tiiese 
characteristics for several eligible groups: 
youti), adults ages 22 to 54, adults ages 22 to 
64, all adults, and adults ages 22 to 64 who 
were unemployed. We determined eligibility 
based on the income of the household and 
tiie JTPA eligibility rules. Ttiie household in- 
come was less than the poverty level or less 
than 70% of the lower level living standard 
or any person in tiie household received wel- 
fare, all persons in tiie household were 
treated as eligible * For each SDA tiie charac- 
teristics of the eligible population were calcu- 
lated for a group of counties tiiat roughly 
corresponds to the counties served by the 
SDA.** 

We correlated the different adult group 



*The entire household was considered a single unit even if it caitained subfainilies. There were some excep- 
tions to this general rule. Unrelated individuals living in a household were treated as separate individuals and not as 
part of the household. Inmates were excluded from the eligible population as were individuals living in college dor- 
mitories; other persons living in group quarters were included 

♦♦The census sample tspos do not generally identify individual counties but rather groups of counties. In some 
cases the county group is a single county or even a large city or subarea within a county. In most cases» however, it 
is a collection of several counties. Thus, it was not possible to calculate the characteristics of the eligible peculation 
for the exact areas served by each SDA. County groups in which a large pwjentage of the population was in coun- 
ties served by an SDA were allocated to that SDA. Several county groups were allocated to more than one SDA, 
and a few were not allocated to any SDA because only a small portion of the county group was served by any single 
SDA. Characteristics were not calculated for a few SDAs because a reasonable qjproximation could not be made. 
Because of the geogr^hic approximations used, these data do not provide good estimates of the size of the eligible 
population for an SDA, but they should provide good estimates of the percentage of the population with various 
characteristics. 
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characteristics with the characteristics of the 
population served by JTPA programs and 
found that the characteristics of adults ages 
22 to 54 had the highest correlations with the 
characteristics of JTPA participants. Thus, 
we used the characteristics of this group in 
our analysis of adult clients to control for dif- 
ferences in the eligible population across 
SDAs.* 

Data on local economic conditions were 
calculated using sources that provide data for 
counties, includmg the ES 202 Program and 
the Local Area Unemployment Statistics 
Program (both in the BLS). We used 
economic conditions that were found to af- 
fect SDA performance in the DOL adjust- 
ment models. 

New Data Collection Strategies 
State Data. We collected information 
about State performance-standards and re- 
lated policies by reviewing written State 
policies for PY 86. We requested from all 
States (and the District of Columbia) the fol- 
lowing types of policies: 

• Incentive and sanction policies; 

• Adjustment policies — whether the 
DOL model is used, description of 
other procedures used to set stand- 
ards, policies regarding adjustments 
to the model-adjusted standards; 

• Additional performance standards 
beyond the seven Federal measures; 
and 

• Policies for serving the hard to serve. 
All States and the District of Columbia 

cooperated with this study and provided us 
with their policies, resulting in 51 observa- 
tions. 

Based on a review of these policies, we 
coded several dimensions of performance- 
standards and hard-to-serve policies. The 



coding fomi for State policies is presented in 
Appendix E. We contacted approximately 
25% of the States by telephone because docu- 
ment review was not sufficient to code the 
policy dimensions that were used in the 
analysis. 

SDA Data. Many of the data required for 
the quantitative evaluation come from the 
SDA level, including data on services, con- 
tracting arrangements, criteria for selecting 
clients, and characteristics of PICs, LEOs, 
and SDAs. 

The new SDA-level quantitative data 
were obtained through a mail survey of all 
SDAs by using two structured question- 
naires. The first questionnaire asked about 
the concerns and influences of the PIC and 
LEO, SDA*s intake procedures, contracting 
procedures, and YEC requirements. We re- 
quested that the director complete this ques- 
tionnaire. The second questionnaire asked 
about factual expenditure and participant data 
from the SDA*s management information 
system (MIS). We requested that this fis- 
cal/MIS questionnaire be completed by the 
staff members who have access to the best in- 
formation on these items. Copies of the direc- 
tor and fiscal/MIS questionnaires are also 
included in Appendix E.** 

Because high response rates were neces- 
sary to ensure that the results were repre- 
sentative of the entire JTPA system, we 
conducted telephone interviews with direc- 
tors who did not respond to the mail survey. 
The detailed factual information in the fis- 
cal/MIS questionnaire was not amenable to 
telephone interviewing, butnonrespondents 
were reminded by telephone to complete the 
questionnaire. 

The response rates to both questionnaires 
were high. Of 610 questionnaires sent out, 
we received 530 responses to director ques- 



*For the analysis of adults over 55. we used the characteristics of all eligible adults. 
**In single-SDA States, similar survey instruments were used but were administered by NCEP staff by 
telephone to tailor the questions to the specific circumstances of each State. 
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tionnaires (87% response rate) and 454 
responses to fiscal/MIS questionnaires (74% 
response rate). 

Table 6 presents a comparison of the 
JASR information and some local characteris- 
tics for three groups: the full sample of 610 
SDAs, the sample of 530 SDAs that 
responded to the director questionnaire, and 
the sample of 441 SDAs that responded to 
both the director and fiscal/MIS question- 
naires. This comparison indicates only minor 
differences in the characteristics of clients 
and local economies between the full sample 
of SDAs and the samples that responded to 
the questionnaires. Thus, the analysis 
samples are very representative of all SDAs 
in the JTPA system. 

A summary of the new information col- 
lected from States and SDAs for the quantita- 
tive analysis is presented in Exhibit 1. 

Analysis Strategy 

The primary objective of the analysis is 
to estimate the relationships specified in the 
conceptual framework presented earlier.* In 
the conceptual framework, dimensions of per- 
formance standards, other policies, and the 
local environment are hypothesized to in- 
fluence program-design decisions; in turn, 
program-design decisions are hypothesized 
to influence the characteristics of the clients 
served, the types of services provided, and 
the costs of those services. 

Two groups of models were estimated in 
the analysis. In the first group, the dependent 
variables were the major outcomes for the 
evaluation: the characteristics of clients 
served, the types of services received, and 
the cost of those services. The objectivf/ of 



these models was to determine (1) the impact 
of State performance-standards policies on 
these outcomes and (2) the impact of SD A 
program-design elements on these major out- 
comes. In the second group of models, the de- 
pendent variables were those program-design 
elements found to affect clients, services, and 
costs. The objective of these models was to 
determine which of these important SDA 
program-design elements are mechanisms 
through which performance standards affect 
the final outcomes. Each of these two groups 
of models is discussed below. 

Models for Clients Served, Types 
of Services Received, and Costs 
In this group of models, the dependent 
variables are (1) the percentage of the SDA's 
terminees with various characteristics, (2) the 
percentage of terminees who received 
various types of services, (3) the average 
length of stay in the program, (4) the Title 
n-A program costs per terminee, and (5) the 
expenditure rates for Title 11- A resources. 
We estimated three types of models with 
these outcome measures. 

The first model estunated the impact of 
State performance-standards policies on 
clients, services, and costs, controlling for 
the other factors that influence program 
design ** The independent variables in this 
model included 

• Dimensions of State performance- 
standards policies 

- Procedures for adjusting beyond 
the model, 

- Emphasis on exceeding standards 
in the 6% policy. 



♦The estimation model, however, involved modification of the conceptual framework for two reasons. First, as 
we indicated earlier. Federal perfomiance-standards policies and other Federal requirements of JTPA do not vary 
among SDAs and thus could not be included in a cross-sectional analysis. Second, some of the variables included in 
the conceptual model were not available for the evaluation. For example, the characteristics of applicants were not 
used because the cost of collecting that information was prohibitive. 

**This model represents the complete reduced form of the recursive system specified in the conceptual 
framework. 
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Comparison of Analysis Samples and Full Sample of SDAs on 
Selected Terminee Cliaracteristics and Local Economic Conditions 



Sampie Responding To 



Full Sample 



Director 
Questionnaire 



Director and 
Fiscal/MIS 
Questionnaires 



Terminee Characteristics 

Percentage of adults who are 
Welfare recipients 
Dropouts 
Minorities 
Females 

Limited-English speakers 
Handicapped 
Offenders 
Ages 55 or older 

Percentage of youth who are 
Welfare recipients 
Dropouts 
In school 
Minorities 
Females 

Limited-English speakers 
Handicapped 
Offenders 
Ages 18 to 21 



29.3 
24.7 
35.6 
55.3 
3.5 
9.6 
8.4 
3.5 



23.2 
24.9 
39.6 
41.8 
50.7 

2.2 
15.9 

7.1 
61.0 



29.6 
24.6 
35.0 
54.9 
3.3 
9.5 
8.6 
3.6 



23.4 
24.6 
40.3 
41.2 
50.4 

2.0 
16.0 

7.2 
60.3 



29.5 
24.4 
34.6 
54.6 
3.2 
9.2 
8.4 
3.7 



23.2 
24.5 
39.9 
40.9 
50.4 

2.0 
15.6 

7.1 
60.8 



Local Economic Conditions 

Unemployment rate 

% fanulies with income below 

poverty 
Average wa^e (in 1,000's) 
% of population in urban 

areas 

Number of SDAs 



7.4 

9.7 

18.2 

64.5 
610 



7.4 

9.7 
18.0 

63.8 

530 



7.5 

9.8 
18.0 

63.1 

441 
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Exhibit 1 

Description of Mormation Obtained in Survey of SDAs and States 

SDADATA 

Director Questionnaire 

PIC Characteristics: 

Concerns of PIC about needs of clients, employers, performance. 
Influence of PIC In types of clients, services, YECs, contract 
types, contract temis, selection of providers. 

LEO Characteristics: 

Concerns of LEO about needs of clients, employers, performance. 
Influence of LEO in types of clients, services, YECs, contract 
types, contract ?erms, selection of providers. 

Performance-Standards Issues: 

Influence of performance standards overall and by measure. 

Adequacy of adjustments. 

Understanding of model 

Experience with adjustments beyond the model 

Intake Procedures: 

Whether SDA or provid'^rs conduct outreach; assignment of 
participants to activities. 
Recruitment difficulty. 

Percentage of eligible applicants receiving services. 
Whether SDA targets services to specific groups; which groups. 
Enrollment niteria by program activity, includng basic 
skills prerequisites, other assessments, employer selection. 

Program Services: 

Types of YECs established 
Qiteria for deficiency. 
Criteria for attainment 

Service-Provider Arrangements: 

Use of performance-based contracts. 

Whether terms vary by client, services, type of provider. 

Payment points. 

Proportion of full price for placement 
Wage requirement 
Placement requirements. 
Client group requirements. 



55 

36 



Design of the Quantitative Evaluation 



SDA Characteristics: 

Proportion of staff worked in CETA. 

Entity that is grant recipient; administrative entity. 

Fiscal/MIS Questionnaire 

Resources available (allocation, carry over, 6% funds). 

Total expenditures. 

Expenditures on administrative costs. 

Expenditures on support costs. 

Expenditures on youth costs. 

Expenditures in performance-based contracts. 

Service provided by type of provider. 

Other program expenditures. 

Incentive fimds received. 

Final performance standards. 

Number of participants receiving job-search assistance, basic 
skills remediation, classroom training in occupational skills, 
OJT, work experience, exemplary youth programs. 

STATE POLICIES 

Use of adjViStment models. 

Policies tor adjustments beyond nxxlel. 

Additional standards. 

Other hard-to-serve policies for specific groups. 
Use of 6% for hard to serve. 
Proportion of 6% used for incentives. 
Standards required to (qualify for incentives. 
Weights for each standard. 

Percentage of incentives received for marginally exceeding 
standards. 

Caps on incentives received. 
Whether service to hard-to-serve groups included in 6% 
calculations. 

Whether expenditure rates included in 6% calculations. 
Whether 6% funds covered by performance standards. 
Definition of failing standards. 
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— Use of 6% for serving the hard to 
serve, and 

— Policies for serving hard-to-serve 
groups; 

• PIC concerns and influence; 

• LEO concerns and influence; and 

• Characteristics of the local environ- 
ment, including the local eligible 
population. 

The purpose of these models was to deter- 
mine the total effects of the State perfor- 
mance-standards policies on clients, services, 
and costs. 

The second model was identical to the 
first except that the program-design elements 
were also included as predictors, including: 

• Process and criteria used to select 
clients and 

• Service-provider arrangements. 
The purposes of this second model were 

(1) to determine the impact of these SDA 
program-design elements on clients, services, 
and costs and (2) to examine which of these 
design elements are potential mechanisms 
through which performance standards in- 
fluence clients, services, and costs.* 

The third type of model estimated the 
relationship among the outcomes of the 
evaluation. We could not estimate simul- 
taneous models among clients, services, and 
costs, so we relied on the qualitative analysis 
to indicate the most common direction of 
causation.** As we discuss in detail in sub- 
sequent chapters, the relationships among 
clients, services, and costs are complex, and 
th^. direction of causation is different in the 



planning stage than in the operational stage. 
In our site visits, we found Uiat typically 
SDAs first identified specific client groups 
that were to be served and then designed 
program services for those clients. Operation- 
ally, however, SDAs funded specific 
programs and the nature of those services af- 
fects the type of clients that enrolled in those 
programs. 

Thus, in our quantitative analysis we in- 
cluded the client groups targeted by the SDA 
as predictors of the services offered and the 
actual percentage of services in various 
program activities as predictors of the types 
of clients actually enrolled in JTPA pro- 
grams. We then included the percentage of 
services in various activities and the percent- 
age of clients with various characteristics as 
predictors of program costs. 

Models for the Elements 
of Program Design 

In the second group of models, elements 
of program design were the dependent vari- 
ables. The objective of estimating these 
models was to examine directly which ele- 
ments of program design are mechanisms 
through which State performance-standards 
policies affect the final outcomes and to fur- 
ther understand the relationship between 
local characteristics and design decisions, 
which will be important background for the 
qualitative evaluation. 

Models were estimated only for those 
program-design elements that were found to 
influence clients served, services received, or 
costs. As indicated by the conceptual 
framework, the independent variables for 



♦Because there were substantially fewer responses to the fiscaVMIS questionnaire than to the director question- 
naire, this model was actually estimated in two steps. First, the effects of the process and criteria used to select 
clients was estimated usin? the sample of SDAs that responded to the director questionnaire. Second, we estimated 
the full model on the sample of SDAs that responded to both the director and flscal/NDS questicmnaires. 

**To estimate the simultaneous system without bias, we need an identifying variable, that is, at least one vari- 
able that affects clients served that does not also affect services received. We were not able to indicate any iden- 
tifiers, and no convincing identifiers have been recommended for estimating a simultaneous system in earlier design 
reports. We emphasize, however, that the first and second models described above are appropriate regardless of the 
causal relationship between clients and services. 
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these models included the dimensions of per- 
formance standards, the characteristics of the 
PIC and LEO, and the characteristics of the 
local environment. 

Estimation 

The models were estimated using multi- 
ple regression analysis*, which yields the in- 
dependent effect of each variable, controlling 
for the effects of all the other variables in the 
model. The numbers presented in the tables 
in Chapters V and VI arc these estimated im- 
pacts. Also presented in the tables are the 
standard errors of the estimated impacts, 
which indicate how precisely the impacts are 
estimated. The larger the impact relative to 
its standard error, the more reliable are the es- 
timated impacts. Those that are very reliable 
are statistically significant, and these effects 
are starred in the tables. 

Statistical Power of the Estimated 
Effects of State and SDA Policies 
The quantitative analysis estimated the 
impacts of State performance-standards 
policies and of different SDA program- 
design elements. It is important to investigate 



whether the power of the quantitative 
analyses is sufficient to detect the effects of 
State and SDA policies and practices. The 
power of analysis depends on the size of the 
sample, on the variation in the dependent and 
independent variables, and on the correlation 
among the variables. 

The example on the next page shows the 
probability of detecting statistically sig- 
nificant impacts on each of the two selected 
outcomes for adults: a $500 impact on cost 
per terminee and a 2.5 percentage point im- 
pact on the percentage of dropouts served. 
We consider four explanatory variables as ex- 
amples: whether the State uses theDOL 
regression model, whether the State requires 
the use of 6% funds for hard-to-serve clients 
(by having a separate pool of 6% funds, by 
requiring 6% be used for the hard to serve, or 
by exempting 6% from standards if used for 
the hard to serve), whether the SDA targets 
dropouts, and the extent to which perfor- 
mance-based contracts are used.** The es- 
timated statistical power of the analyses of 
the relationships between these explanatory 
and outcome variables are as follows.*** 



♦We used ordinary least squares (OLS) to estimate the empirical models. In most cases, the dependent variable 
is continuous and OLS is an entirely appropriate estimation technique. Although some of these continuous variables 
are limited in range (for example, percentage of clients who arc high school dropouts, pm:entage of clients who 
received OJT), these limits are generally not constraining and do not significantly affect the distribution of the error 
term. Thus, OLS is appropriate even though the dependent variable is limited in range. In some cases, the dependent 
variables are binary, so that OLS models are inefficient because the error term is necessarily heteroscedastic and can 
yield predicted probabilities outside the range from zero to one. Generally, however, our experience with alternative 
techniques such as logit or probit has been that they yield results very similar to OLS unless the mean of the depend- 
ent variable is close to one or zero. 

**The effects of the use of the model, the use of 6% for hard to serve, and the SDA targeting dropouts are es- 
timated in the director questionnaire sample, while the eifects of the use of performance-based contracts is based on 
the fiscal/MIS questionnaire sample. 

***The calculation is based on a .05 significance level and the actual standard errors in the models estimated 
for the quantitative analysis. 
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Use of DOL model (effect of all 

standards vs. no standards) 
Use of 6% for hard to serve 
Whether the SDA targets dropouts 
Use of performance-based contracts 

(percent, effect of 50% change) 

These results indicate that the power of 
the quantitative analysis is substantial (over 
90%) for policies that vary considerably 
across States. Because a large majority of the 
States use the DOL models, the power of the 



Cost per 

Terminee 

($500) 

42% 
99% 



99% 



Percentage of 
School Dropouts 
(2.5) 

32% 
93% 
99% 

95% 



analysis of the impact of using the model is 
lower. Overall, the power of the quantitative 
analysis is sufficient to detect effects of 
performance-standards policies. 
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\. Impact of Policies and Practices on 
Clients Served in JTPA Programs: 
Evidence from the Quantitative Analysis 



Introduction 

In this chapter we examine, using quan- 
titative data, the impact of State performance- 
standards policies and other selected State 
and SDA policies and practices on the types 
of clients served by JTPA programs. Specifi- 
cally, we examine whether differences in 
policies and practices are associated with 
variation in the characteristics of clients 
served, as reported in the JASR. We inves- 
tigate the effects of policies and practices on 
clients often viewed as hard to serve, includ- 
ing welfare recipients, dropouts, minorities, 
females, and those with other barriers to 
employment (including handicapped, of- 
fenders, and those with limited English- 
speaking ability). For adults, we also 
examine those 55 years old and over, and for 
youth, we examine in-school youth and those 
18 to 21 years old. The average percentages 
of terminecs served with these characteristics 
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are presented in Table 7 for adults and youth. 

Historical data from job-training 
programs show that, witii the exception of 
the older youth, participants with one or 
more of tiiiese characteristics are hard to serve 
in that they often require more intensive train- 
ing and more supportive services and they 
have a harder time achieving a positive out- 
come than the other participants. As a result, 
most of these characteristics are included in 
the optional DOL adjustment models for 
PY 86. (The exception is the other barriers 
to employment because these characteristics 
either were not available for use in the 
models or were too rare to estimate reliably 
their impact on performance.) 

When examining the impact of JTPA 
policies and practices, we control for differen- 
ces among SD As on many dimensions, the 
most important being the characteristics of 
the eligible population. As described pre- 



Table 7 



Means of Client Characteristics for JTPA Terminees 
and Eligible Populations in SDA 



Adults 



JTPA Eligible* 
Terminees Population 



Welfare Recipients 


29.5% 


30.6% 


Dropouts 


24.6 


37.3 


Minoritiss 


^5.0 


26.4 


Females 


54.9 


56.6 


Handicapped 


9.5 


15.3 


Offenders 


8.6 




Limited-English speakers 


3.3 


3.3 


Ages 55 and older 


3.6 





^e average characteristics of the eligible population ages 22 to 54. 



Youtk 





PY 86 JTPA 
Terminees 


Eligible 
Population 


Welfare Recipients 


23.4% 


32.5% 


Dropouts 


24.6 


15.8 


In school 


40.3 


46.5 


Minorities 


41.2 


30.9 


Females 


50.4 


53.2 


Handicapped 


16.0 


3.7 


Offenders 


7.2 




Limited-English speakers 


2.0 


1.9 


Ages 18 to 21 


60.3 


53.2 
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Impact of Policies and Practices on Clients 



viously, we calculated these characteristics 
from the Census Bureau sample for each 
SDA. The averages of these characteristics 
are also presented in Table 7. 

Comparison of the characteristics of the 
eligible population to those of JTPA ter- 
minees shows a pattern that has been found 
in other studies. Specifically, for adults, 
JTPA programs tend to serve welfare 
recipients and women in proportion to their 
incidence in the eligible population but tend 
to serve proportionately more minorities and 
fewer dropouts than are in the eligible popula- 
tion. For youth, JTPA programs tend to serve 
in-school youth and females in proportion to 
their incidence in the eligible population but 
serve more dropouts and minorities and 
fewer welfare recipients than in the eligible 
population. 

However, even in those cases in which 
the characteristics of the eligible population 
over- or understate the characteristics of 
JTPA participants, the variation across SDAs 
in the characteristics of eligible population is 
a very strong predictor of variation in the 
characteristics of the individuals served. 
Thus, the analysis controls well for important 
differences among SDAs in the characteris- 
tics of the eligible population that indepen- 
dently affect the types of clients served. 

Overview of Chapter 
and Key Findings 

The analysis in this chapter is organized 
following the conceptual framework 
presented in Chapter H. We begin by examin- 
ing the impact of State and local factors that 
may influence SDA design decisions and 
thus the types of clients served, including 

• State performance-standards 
policies— We find that State 6% 
policies that emphasize exceeding 
rather than just meeting standards and 
policies that place greater weight on 
the cost standard tend to discourage 
service to several hard-to-serve 



groups. State policies that adjust 
standards for serving hard-to-sorve 
groups and policies identifying State 
priorities for client groups tend to en- 
courage service to hard-to-serve 
groups. 

• PIC and LEO concems and in- 
fluence —We find that PICs that are 
more influential in program design 
tend to be associated with greater ser- 
vice to some hard-to-serve groups 
while PICs more influential in con- 
tracting tend to be associated with 
less service to hard-to-serve groups. 

• Local economic conditions — Al- 
though many characteristics of the 
local area are controlled for, we 
present only those economic condi- 
tions included in the JTPA allocation 
formula. We find that local unemploy- 
ment rate has a strong influence on 
the types of clients served: SDAs in 
areas with high unemployment rates 
are less likely to serve hard-to-serve 
clients than those in areas with low 
unemployment. 

Next, we add SDA practices that may 
also influence the types of clients served, 
including 

• Enrollment criteria^ We find that 
SDAs that use objective criteria serve 
more hard-to-serve clients than SDAs 
that rely on subjective judgixients. 

• Service-provider arrangements — We 
find that service-provider arrange- 
ments have fewer effects than ex- 
pected. In particular, use of 
performance-based contracts has little 
effect on the types of clients served. 
We do find that SDAs that vary the 
terms of contracts serve more hard-to- 
serve clients than those that impose 
the same terms on all providers. 

Then, we examine the relationship be- 
tween clients served and services provided 
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by adding 

• Types of program services— We find 
a clear pattern where basic skills 
remediation has the strongest associa* 
tion with hard-to-serve clients, fol- 
lowed by classroom training in 
occupational skills, job-search assis- 
tance, and finally OJT* 

The final section examines whether SDA 
practices that were found to affect the types 
of clients served were themselves influenced 
by State performance-standards policies. We 
find that 

• State performance-standards policies 
affect SDA practices, which in part 
explam the effects of performance- 
standards policies on the types of 
clients served in JTPA programs. 

Impact of State Policies 

The estimated impacts of State perfor- 
mance-standards and related policies on the 
percentage of clients with different charac- 
teristics are presented in Table 8 for adults 
and Table 9 for youth.* In estimating the im- 
pacts of State-level policies, the analysis is 
limited by the fact that there are only 5 1 uni- 
que combinations of those policies. Policies 
used by only one or two States may serve as 
proxies for some unique circumstances in 
those States so it is difficult to obtain reliable 
estimates of the independent effects of rarely 
used policies. Thus, we have combined some 
of the policies with related policies to obtain 
greater variation across States.** 

The numbers presented in Tables 8 and 9 



indicate the independent effects of each 
policy or procedure. These effects represent 
the change in the percentage of clients with 
specific characteristics (for example, welfare 
recipients) for each unit change in the policy 
or practice. 

The units in which the policies or prac- 
tice are measured depend on the policy. 
Some policies we measure whether or not the 
State has that policy, for example, whether or 
not the State has a policy to use 6% funds for 
the hard to serve. Table 8 indicates that the 
effects of this policy on welfare recipients is 
4.67 (fourth row, first column). This result 
indicates that SDAs in States with such a 
policy serve 4.67 percentage points more wel- 
fare recipients than SDAs in States without 
such a policy. Other policies are measured 
with a continuous variable, for example, the 
incentive weight placed on the cost standard 
is measured as a percentage. The estimated 
effect of this variable on welfare recipients is 
-0.43. Thus, for each percentage point in- 
crease in the weight on the cost standard, the 
percentage of welfare recipients is estimated 
to fall by 0.43 percentage points. Thus, 
SDAs in States with a weight of 15% on the 
cost standard are estimated to serve 4.3 per- 
centage points more welfare recipients than 
SDAs in a State with a weight of 25% on the 
cost standard. 

Presented also are the standard errors of 
the estimates, which indicate how reliably 
the impacts are estimated. Those impacts that 
are large relative to their standard errors and 
thus most reliable are statistically significant 



♦The impacts of State policies are estimated controlling for PIC and LEO concerns and influence, as well as the 
characteristics of the eligible population and the local area. TTie fraction of the variation in the dependent variables 
(R s) explained by these initial models are generally quite high. For adults, the R^s are .58 for welfare recipients, 
.47 for dropouts, .88 for minorities, .28 for females, .37 for other bamers, and 22 for those age 55 and older. For 
youth, the R s are .49 for welfare recipients, . 19 for dropouts, . 14 fw in school, .87 for minorities, .24 for females. 
.33 for other barriers, and .15 for those age 18 to 21 . 

**In our preliminary analysis, we also examined the impacts of other State policies, including the level of 
which standards are set (for example, lower tolerance range). State post-program standards, and dimensions of sanc- 
tion policies. These policies had inconsistent effects that were not reliably estimated and so were not included in the 
final models. 
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Table 8 

Estimated Impacts of State Policies on Percentage of JTPA Terminees "ivith 

Various Characteristics — Adults 

(Standard Errors in Parentheses) 





Welfare 
Recipients 


Dropouts 


Minorities 


Females 


Other Barriers to 
Employment 


55 Years and 
Older 


Useof nuxlel 
(number of standards 
for which model is used) 


1.10*** 
(0.32) 


0.54** 
(0.22) 


0.38 
(0.33) 


0.42 
(0.27) 


0.01 
(0.33) 


-0.11 
(0.11) 


Adjustment procedures 
specified in policy 


0.57 
(1.06) 


0.44 
(0.68) 


1.84* 
(1.04) 


0.12 
(0.88) 


-3.04*** 
(1.03) 


-1.73 
(0.37) 


Eivq)hasis on exceeding 
standards in 6% policy 


-1.54** 
(0.63) 


0.19 
(0.42) 


-1.34** 
(0.64) 


0.77 
(0.55) 


-0.16 
(0.64) 


-0.37* 
(0.23) 


use or yyfo for 
hard to serve 


fl ^\ # 

(l'.09) 


-0.74 
(0.75) 


-U.oo 
(1.28) 


1 «)k 
-1.00^ 

(LOO) 


(1.12) 


f\ f\f\ 

-0.00 
(0.40) 


State policy for 
serving client group 


(U3) 


2.26*** 
(0.82) 


1.06 
(1.28) 


0.18 
(1.06) 


0.09 
(1.18) 


1.78*** 
(0.44) 


Incentive weiglit on 
cost-per-entcred- 
employment standard (%) 


.0.43*** 
(0.14) 


-0.16** 
(0.06) 


0.04 
(0.08) 


-0.09 
(0.07) 


0.00 
(0.08) 


-0.01 
(0.03) 


Incentive weight on 
wage standard (%) 


0.17 
(0.12) 


-0.11* 
(C.07) 


-0.12 
(0.10) 


0.06 
(0.09) 


0.27** 
(0.10) 


0.09** 
(0.04) 


Incentive weight on 
welfare standmd (%) 


0.09 
(0.17) 













* Statistically significant at .10 level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 



Table 9 

Estimated Impacts of State Policies on Percentage of JTPA Terminees with 

Various Characteristics— Youth 

(Standard Errors in Parentheses) 





Recipients 


Tii¥\tv%i 1 f e 
l/lV/pUUU> 


in ocnooi 


Minorities 


Females 


Other Barriers to 
Employment 


Ages 
18 to 21 


Use of model 
(number of standards 


0.16 
(0.31) 


0.43 
(0.39) 


-0.07 
(0.60) 


0.03 
(0.38) 


0.09 
(0.22) 


0.91** 
(0.45) 


-0.43 
(0.53) 


Adjustment iMtxy^dures 

opvCiii^l 111 pi/ilcy 


3.14*** 
U.uo) 


1.19 
(1.28) 


0.63 
(2.19) 


2.27** 
(1.25) 


0.79 
(0.72) 


-2.00 
(1.46) 


0.76 
(1.74) 


Emphasis on exceeding 
standards in 6% policy 


-1.46*** 
(0!60) 


-0.75 
(0.74) 


-3.56*** 
(1.20) 


-2.65*** 
(0.71) 


0.50 
(0.42) 


-0.62 
(0.85) 


5.00*** 
(1.00) 


Use of 6% for 
hard to serve 


4.72*** 
(1.16) 


-5.30*** 
(1.41) 


10.02*** 
(2.29) 


1.68 
(1.49) 


-1.91** 
(0'.86) 


1.69 
(1.64) 


-7.80*** 
(1.94) 


State policy for 
serving client group 


0.85 
(1.00) 


1.05 
(1.53) 


0.04 
(4.91) 


1.31 
(1.41) 


-0.35 
(0.85) 


0.38 
(1.59) 




Incentive weight on 
cost-pcr-positive- 
termination standard (%) 


-0.27*** 
(0.08) 


-0.28*** 
(0.12) 


0.16 
(0.17) 


0.1? 
(0.10; 


-0.08 
(0.06) 


-0.33*** 
(0.12) 


0.06 
(0.14) 


Incentive weight on entcred- 
employment-rate standard (%) 


-0.14* 
(0.08) 


0.24*** 
(0.09) 


-0.33** 
(0.14) 


-0.07 
(0.09) 


-0.01 
(0.05) 


-0.18* 
(0.11) 


0.17 
(0.13) 
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* Statistically significant at .10 level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 
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(Throughout this report the term "significant" 
is used in this technical sense.) Because we 
are examining impacts of a large number of 
policies and practices, some significant ef- 
fects could occur by chance. TTius, in our in- 
terpretations of the funding, we place greater 
emphasis on the impacts that are consistent 
over several client groups or between adults 
and youth. 

Results for Adults 

The DOL models are intended to allow 
SDAs to serve hard-to-serve clients by adjust- 
ing SDAs' performance standards for the 
characteristics of clients served. The results 
for adults indicate that the number of stand- 
ards for which the DOL adjustment models 
are used significantly increases the percent- 
age of welfare recipients and dropouts served 
by JTPA programs. These results indicate 
that SDAs in a State that used the model for 
all seven standards would serve 7.7 percent- 
age points* more welfare recipients and 3.8 
percentage points more dropouts than SDAs 
in a State where the models were not used for 
any standards. Thus the adjustment models 
appear to have their intended affects of allow- 
ing SDAs to serve these harder-to-serve 
groups and still meet their performance stand- 
ards.** 

As indicated in Chapter III, approximate- 
ly one-third of the States also specify proce- 
dures for SDAs to obtain additional adjust- 
ments beyond the model. Having a State 
policy that specifies procedures for obtaining 
adjustments beyond the model significantly 
increases services to minorities but has unex- 
pectedly negative impacts on those with 
other barriers to employment (handicapped. 



offenders, or those with limited English) and 
those age 55 and older. This policy may be 
serving as a proxy for other conditions in 
those States, but it is also possible that adjust- 
ments beyond the model are made for service 
to other groups (for example, displaced 
homemakers) at the expense of service to 
groups listed in the JASR. 

States vary widely in the emphasis their 
6% policies place on SDAs exceeding their 
performance standards. The third row of 
Tables 8 and 9 indicates the estimated effects 
of the emphasis that 6% policies place on 
SDAs exceeding their standards, an index of 
several policies. Our preliminary analyses in- 
dicated that four State policies were strongly 
correlated: (1) giving only a small amount of 
incentives for marginally exceeding stand- 
ards, (2) competition among SDAs for incen- 
tive funds, (3) not having a cap on perfor- 
mance (beyond which additional incentives 
would not be earned), and (4) the number of 
standards that must be exceeded to be 
eligible for incentives. Because of this cor- 
relation, we were not able to estimate reliably 
the impacts of these policies separately and 
instead combined them into an index repre- 
senting the overall emphasis on exceeding 
standards.*** 

3DAs in States with policies that place a 
strong emphasis on exceeding standards 
serve significantly fewer welfare recipients, 
minorities, and older adults. Although these 
effects are highly significant, they are not 
large. For example, SDAs in States that score 
low on this index serve 2.4 percentage points 
more welfare recipients than SDAs in States 



♦That is, 7x1.10. 

**Only eight Slates do not use the DOL models. Three are single-SDA States, and several are less densely 
populated States. The analysis controls for the effects of being single-SDA States and of population density. The ef- 
fects of the model, therefore, are independent of the effects of these other characteristics. 

***The index was constructed using factor analysis, which examined the correlations among these policies. 
The analysis indicated (unstandardized) weights for each policy: percentage of incentives received for marginally ex- 
ceeding standards, -.0098; placing a cap on performance, 0.72; having competition among SDAs, 0.59; number of 
standards that must be exceeded, 0.03. 
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that score high on this index * 

States also use performance-standards 
policies to further State goals for the types of 
clients to be served in JIx'a programs. 
Several States have 6% policies that em- 
phasize services to the hard to serve, either 
by setting aside a pool of funds at the State 
level, requiring SDAs to use some 6% funds 
for the hard to serve, or exempting 6% funds 
from performance standards if they are used 
for the hard to serve.** Those 6% polic^^s 
that emphasize service to the hard to serve 
tend to increase the percentage of welfare 
recipients and adults with other barriers to 
employment. 

These policies, however, are associated 
with a significant decrease in the percentage 
of women served, a result that is particularly 
surprising given the large increase in welfare 
recipients served. Further analysis indicates, 
however, that SDAs in States with general as- 
sistance programs are particularly likely to 
reduce the percentage of women served in 
response to these hard-to-serve policies. In 
the qualitative section we discuss how some 
SDAs balance service to specific hard-to- 
serve clients with increased service to other 
easier-to-seive clients as a strategy for meet- 
ing performance-standards goals. 

The fifth row in Tables 8 and 9 presents 
the impact of having a State policy for serv- 
ing a specific client group, defined as having 
a State standard for service to that group, in- 
tegrating service to that group into the cal- 
culation of incentive grants, or identifying 
the specific group as a priority group.*** 
(The results presented in Tables 8 and 9 rep- 



resent the effect of having a policy for wel- 
fare recipients on the percentage of welfare 
recipients served, the effect of a policy for 
dropouts on the percentage of dropouts 
served, and so on.) Having State policies for 
service to welfare recipients, dropouts, or 
older adults results in large and significant ef- 
fects on those client groups served by JTPA 
programs. 

The fmal State performance-standards 
policy that we examine is the weights that 
are placed on the various Federal standards 
in calculating incentive grants. These results 
provide some evidence about the relative im- 
pacts of these Federal standards on the types 
of clients served. Some States explicitiy 
place different weights on the different stand- 
ards. However, States that give equal weight 
to the percentage by which SDAs exceed 
their standards implicitly give more weight 
to the cost standard because it is easier to be 
significantiy below the cost standard than, 
for example, significantly above the wage 
standard, as discussed in Charjter HI. 

Placing greater weight on the cost stand- 
ard significantly reduces service to welfare 
recipients and dropouts. For example, SDAs 
in States that place a low weight of 15% on 
the cost standard serve 4.3 percentage points 
more welfare recipients than SDAs in States 
that place a high weight of 25% on the cost 
standard. 

Placing a high weight on the wage stand- 
ard also reduces service to dropouts but sig- 
nificantiy increases tiie percentage of clients 
over 55 or with otiier barriers to employment 
(handicapped, offenders, or those with 



* An example of a State policy that scores low (25lh percentile) on this index is one that requires only four 
standards be exceeded, gives 20% of the incentives for just marginally exceeding, has a cap on performance beyond 
which no further incentives are given, and has no competition among SDAs. An example of a policy that scores 
high (75th percentile) is one that requires six standards be exceeded* gives no incentive payments for marginally ex- 
ceeding, has no cap on performance, and has competition among SDAs. 

**We examined the separate impacts of these policies in preliminary analysis, but their separate impacts were 
less reliably estimated than the summary measure of using any one of these policies. 

♦♦♦Preliminary analysis examined the separate impact of having a standard and of having a policy identifying 
the group as hard to serve. The separate results were Icgc consistent and less significant than those presented here. 
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limited English). The effect on clients 55 and 
older probably occurs because older workers 
have more work (experience and can earn 
higher wages, but the effect on clients with 
other barriers is unexpected. The weight 
placed on the welfare entered-employment 
rate does not significantly affect the percent- 
age of welfare recipients served. 

Results for Youth 

Table 9 presents the estimated effects o 
State policies on the types of youth served. 
The estimated impact of the number of stand- 
ards for which the DOL adjustment models 
are used is considerably smaller than for 
adults, although the use of the models sig- 
nificantly increases the percentage of youth 
with other barriers to employment. Pro- 
cedures for adjusting beyond the model, 
however, tend to have greater effects, sig- 
nificantly increasing the percentage of wel- 
fare recipients and minority youth. As with 
adults, the estimated impact of having these 
adjustment procedures is to reduce the per- 
centage of youth with other barriers to 
employment. 

As was the case for adults, incentive 
policies that place greater emphasis on ex- 
ceeding standards result in significantly 
fewer welfare recipients and minorities being 
served by SDAs. These policies also sig- 
nificantly affect SDAs* decisions about serv- 
ing in-school versus out-of-school youth, 
significantly reducing the percentage of 
youth who are in school, for whom positive 
termination is the major outcome, and in- 
creasing the percentage who are 18 to 21, for 
whom both an entered employment and posi- 
tive termination can be attained. 

Policies that use 6% for the hard to serve 
significantly increase the percentage of wel- 
fare recipients and particularly the percentage 
of in-school youth served. These policies 
I nd to decrease service to dropouts, primari- 
ly because they decrease service to youth 
ages 18 to 21, who ai'e more likely to be 
dropouts than younger youth. As with adults. 



these policies are associated with a sig- 
nificant decrease in service to females. 

A State policy for the specific client 
group is estimated to increase the percentage 
of welfare recipients, dropouts, and 
mmorities, although none of these effects are 
statistically significant for youth. 

Placing greater weight on the youth cost- 
per-positive-termination standard significant- 
ly reduces service to several hard-to-serve 
grouos, including welfare recipients, drop- 
outs, and youth with other barriers to employ- 
ment. Placing greater weight on the 
entered- enii^loyment-rate standard signifi- 
cantly rbviuces the percentage of welfare 
recipients and the percentage of minorities. 
The weight on the entercd-employment-rate 
standard also significantly affects SDAs* 
decisions about whether to serve in-school 
youth, significantly decreasing the percent- 
age of in-school youth. Because many of the 
dropouts are older and in predominandy 
employment-oriented programs, higher 
weight on the entered-employment-rate stand- 
ard also increases service to dropouts for 
youth. 

Summary of Impacts of State Policies 

The results indicate aiat State perfor- 
mance-standards policies can affect the types 
of clients served in JTPA programs. States 
that place a strong emphasis on exceeding 
performance standards in their incentive 
policies tend to reduce service to several hard- 
to-serve groups, including both adult and 
youth welfare recipients. Furthermore, States 
that heavily weight the cost standards in cal- 
culating incentives tend to reduce service to 
Uard-to-serve groups, including welfare 
recipients and dropouts. Awarding incentive 
payments based on the extent to which stand- 
ards are exceeded and the presence of cost 
standards are both requirements in the JfPA 
legislation. 

On the other hand, some performance- 
standards policies encourage service to hard- 
to-serve groups. The use of the DOL 
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adjustment models increases service to 
several hard-to-serve groups, particularly for 
adults. States with procedures for adjusting 
beyond the model tend to encourage service 
to some hard-to-serve groups, although these 
effects arc less consistent. In addition. States 
that require or encourage the use of 6% funds 
for the hard to serve and States that establish 
policies for specific client groups, including 
establishing State performance standards for 
service to client groups, tend to encourage 
service to more hard-to-serve clients in JTPA 
programs. 

Impact of PIC and LEO 
Concerns and Influence 

Next, we examine the impact of PIC and 
LEO concerns and influence on the types of 
clients served in JTPa programs. As indi- 
cated in our conceptual framework, the PIC 
and LEO are hypothesized to influence the 
design of the local JTPA program and thus 
the types of clients served. We examine the 
impact of the extent to which PJCs and LEOs 
are influential in program design, are influen- 
tial in contractual arrangements, p^d arc con- 
cerned about SDA performance. 

In our questionnaires, SDA directors 
v/erc asked to rate the influence of the PIC 
and of the LEO in decisions about the types 
of clients to serve, the types of services to 
provide, the development of YECs, the selec- 
tion of service providers, and the types and 
terms of service-provider contracts. Our 
preliminary analysis of the corrclations 
among the answers to these questions indi- 
cated that therc werc two basic dimensions 
of influence: influence in program design 
(clients, services, and development of YECs) 
and influence in contracting arrangements 
(choice of providers, type of contracts, and 



terms of contracts). Furthermorc, the in- 
fluence of the PIC was not correlated with 
the influence of the LEO, The separate im- 
pacts of the PIC and LEO design and con- 
tracting influence on the types of adult 
clients served in JTPA programs are pre- 
sented in Table 10.* 

Results for Adults 

The results indicate that in SDAs where 
the PIC is influential in program design, 
there is significantly greater service to wel- 
fare recipients, minorities, and females. In 
contrast, in SDAs where the PIC is influen- 
tial in contracting, there is significantly less 
service to welfare recipients. Apparendy, 
PICs that see their role as influencing the 
design of the program favor service to the 
hard to serve while there is some evidence 
that PICs that see their role as watching over 
how JTPA funds are spent favor service to 
other groups. The influence of the LEO had 
small estimated impacts that were not statisti- 
cally significant 

Directors also rated how concerned the 
PIC was about various aspects of SDA per- 
formance, including performance relative to 
the standards set by the State, to the national 
standards, to performance of other SDAs in 
the State, and to last year's performance. 
Similar questions were asked about the LEO, 
Our analysis found that the PIC and LEO's 
emphases on performance were very corre- 
lated, so we combined these sets of answers. 
We separated out the concern about perfor- 
mance relative to the standards set by the 
State and performance relative to other 
criteria because the intention of the perfor- 
mance-standards system is that SDAs be 
judged according to their own standards, not 
against these other criteria. 



♦Although PIC and LEO influences in design are cwrelated with PIC and LEO influences in contracting, these 
correlations only slightly increase the standard errors of the estimates. In our preliminary analysis we also examined 
the impact of PIC and LEO concern about client needs and about employer needs, but these variables had no sig- 
nificant impact and are not included in the final models. The impacts of PIC and LEO concerns and influence are es- 
timated while controlling for State policies and the characteristics of the eligible population and the local area. 
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Table 10 



Estimated Impacts of PIC and LEO Concerns and Influence on Percentage of 
JTPA Terminees with Various Characteristics— Adults 

(Sfndard Errors in Parentheses) 





Welfare 


Dropouts 


Minorities 


Females 


Other Barriers to 
Emolovnient 


55 Years and 
Older 


nC influence on 


1.86** 


0.62 


1.60* 


1.29* 


0.83 


0.13 


program design 


(0.81) 


(0.55) 


(0.83) 


(0.70) 


(0.82) 


(0.29) 


PIC influence in 


-1.27* 


0.19 


0.04 


-0.59 


-0.41 


0.14 


contiBCting 






{{). /V) 


[yj.jy) 




(0 25) 


LEO influence on 


-0.22 


0.51 


0.92 


0.43 


-0.40 


0.23 


program design 


(U.ol) 






(v./yj) 






LEO influence in 


0.06 


0.41 


0.97 


0.07 


0.74 


-0.29 


contracting 


(0.65) 


(0.44) 


(0.67) 


(0.57) 


(0.66) 


(0.24) 


PIC and LEO concern 














about performance 














— Relative to standards 


0.37 


0.49 


-0.92 


-0.56 


-0.42 


-0.01 


set by State 


(0.75) 


(0.51) 


(0.78) 


(0.67) 


(0.77) 


(0.27) 


— Relative to other 


0.06 


-2.18*** 


-1.61* 


0.10 


-1.28 


-0.06 


criteria 


(0.88) 


(0.60) 


(0.90) 


(0.77) 


(0.90) 


(0.32) 



* Statistically significant at .10 level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 
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Quantitative Analysis 



The results in Table 10 indicate that the 
intensity of the PIC and LEO concern about 
performance relative lo the SDA*s own stand- 
ards has small and inconsistent impacts on 
the types of clients served. However, their 
concern about performance relative to the 
other criteria significantly reduces service to 
dropouts and minorities and has a negative 
impact on service to those with other barriers 
to employment, although the latter effect is 
not statistically significant. 

iiesults for Youth 

Table 1 1 presents the estimated impacts 
of PIC and LEO concerns and influence on 
the youth groups served. Similar to the pat- 
tern observed for adults, the estimated impact 
of the PIC being more influential in program 
design is to increase significantly the percent- 
age of youth with other barriers to employ- 
ment, including handicapped, offenders, and 
those with limited English. Although not 
statistically significant, PIC influence on 
design also tends to increase the percentage 
of welfare recipient and in-school youth and 
decrease the percentage of older youth in 
JTPA programs. PICs that are influential in 
contracting, however, are estimated to sig- 
nificantly reduce service to welfare youth, 
and LEOs influential m contracting are es- 
timated to significantly decrease service to 
in-school youth. 

The impact of the PIC and LEO concern 
with performance is s-^mewhat different for 
youth than for adults, in that the concern for 
performance relative to the SDA*s standards 
is associated with significantly fewer female 



youth receiving JTPA services and the con- 
cern about performance relative to other 
criteria is associated with significantly fewer 
older youth being served. 

Summary of Impact of PIC and LEO 

Concerns and Influence 

Clearly, the few questions asked about 
the concern and influence of the PIC and 
LEO only begin to capture the complex 
relationship between these two partners and 
SDA decisions about whom to serve in JTPA 
programs. Nonetheless, the results suggest 
that PIC influence per se does not necessarily 
reduce service to the hard to serve. PICs that 
see their role as guiding the design of the 
program are associated with greater enroll- 
ment of the hard to serve while PICs that see 
their role as guiding the spending of the 
funds and choice of providers are associated 
with somewhat less enrollment of those 
groups. In general, the PIC has a substantial- 
ly greater impact than the LEO. 

The results about PIC and LEO concern 
about performance are far fi'om conclusive* 
However, there is suggestive evidence that 
concern about performance relative to the 
SDA's own standards is less a barrier to serv- 
ing the hard to serve than concern about per- 
formance relative to other criteria, such as 
the perfoimance of other SDAs in the State. 
Several public interest groups have devoted 
considerable effort and resources to educat- 
ing PICs and to some extent LEOs about the 
intention of performance standards and of the 
adjustment process. These results suggest 
that such efforts are well placed. 



i 4 
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Table 11 



Estimated Impacts of PIC and LEO Concerns and Influence on Percentage of 
JTPA Terminees with Various Characteristics— Youth 

(Standard Enors in Parentheses) 





Welfare 


Dropouts 


In School 


Minorities 


Females 


Other Barriers to 
employment 


Ages 
io ro zi 


PIC influence on 


1.18 


-O.U 


1.19 


-0.16 


0.84 


2.04* 


-1.97 


piogram design 


(0.81) 


(1.00) 


(1.57) 


(0.97) 


(0.57) 


(1.15) 


(1.38) 


PIC influence in 


-1.52** 


-0.51 


-0.87 


0.81 


-0.54 


-1.27 


1.56 


contracting 


(0.67) 


/A O A\ 

(O.o4) 


(1.31) 


(0.81) 


(0.47) 


//\ A/'\ 

(0.96) 


(1.15) 


LEO influence on 


0.20 


0.79 


-1.01 


1.12 


-0.07 


-1.20 


0.83 


program design 


(U.ol) 


(1.01) 


(1.58) 


/A AO\ 

(0.98) 


/A cn\ 

(0.57) 


(1.16) 


(1.39) 


LEO influence in 


-0.63 


1.13 


-2.54** 


0.19 


0.20 


0.63 


1.68 


contracting 


(0.65) 


(0.81) 


(1.26) 


(0.79) 


(0.46) 


(0.93) 


(1.11) 


PIC and LEO concern 
















about performance 
















— Relative to standards 


-0.03 


-1.10 


1.09 


-0.59 


-1.03** 


0.63 


0.11 


set by State 


(0.75) 


(0.93) 


(1.46) 


(0.91) 


(0.53) 


(1.07) 


(1.28) 


— Relative to other 


0.73 


-1.06 


2.44 


-0.91 


0.28 


-0.13 


-2.57* 


criteria 


(0.88) 


(1.10) 


(1.71) 


(1.07) 


(0.62) 


(1.26) 


(1.51) 



* Statistically significant at .10 level. 
** Statistically significant at .05 level 
*** Statistically significant at .01 level. 



Quantitative Analysis 



Impact of Local Conditions Used 
in Allocation Formula 

All of the analyses presented in this 
report control for numerous local condi- 
tions.* Of particular interest, however, are 
the unemployment rate and the percentage of 
the fa' lilies with incomes below poverty 
level because these two characteristics are 
used to determine how much JTPA funding 
each SDA receives. Thus, examining the im- 
pact of these local conditions gives some in- 
dication of the impact the Federal allocation 
formula has on the types of clients served in 
JTPA programs. 

Table 12 presents these results for both 
adults and youth. SDAs operating in ai^as 
with higher unemployment rates serve sig- 
nificantiy lower percentages of dropouts, 
minorities, and those with other barriers to 
employment for adults and significantiy 
lower percentages of dropouts and minorities 
for youth. These results are very consistent 
with our findings from the case studies in 
which we found that SDAs in areas of lower 
unemployment were more likely to serve 
hard-to-serve clients with substantial barriers 
to employment who were made aware of 
JTPA programs by vigorous outreach efforts. 
In contrast, SDAs in areas of higher un- 
employment were more likely to serve those 
who were recently unemployed, who had pre- 
vious work histories, and who sought out 
JTPA services on their own. 

The percentage of families with incomes 



below poverty level has less impact on the 
types of clients served, although areas with 
more people below poverty level tend to 
serve fewer adults with other barriers to 
employment, fewer older adults, fewer youth 
with other barriers to employment, and fewer 
youth ages 18 to 21. 

The allocation formula indicates the dual 
purpose of JTPA programs of serving those 
that are structurally unemployed via the 
poverty rate and serving those that are cycli- 
cally unemployed via the unemployment 
rate. These results indicate that basing JTPA 
funding on the unemployment rate increases 
funds to areas that serve clients who are 
more job ready but who are, nonetheless, 
eligible for JTPA services. 

Impact of Enrollment Criteria 

The next step in our analysis is to ex- 
amine the impact of SDA procedures on the 
types of clients served in JTPA programs, in- 
cluding enrolLnent procedures and service- 
provider arrangements. These practices 
represent decisions that SDAs made in the 
design of local JTPA programs. It is impor- 
tant to examine the impact of these practices 
on clients served both because the results can 
be useful in improving program design and 
because these practices may be mechanisms 
thiough which performance standards in- 
fluence clients served, as described in the 
conceptual framework. This section, there- 
fore, examines the impact of SDA practices 
on the types of clients enrolled in JTPA 



♦Included as control variables in aU the analyses are characteristics of the eligible populaUon (percentage of 
dropouts, post-high-school attendees, minorities, welfare recipients, females, those with limited English-speaking 
ability, handicapped, long-tenn unemployed, and. for adults, ages 22 to 29. and. for youth, ages 14 to 15. ages 16 to 
17. and in school), characteristics of the local area (unemployment rate, population density, percentage of families 
with income below poverty level, net commuting into the area, percentage of population living in urban area, and 
average annual earnings in wholesale or retail trade), and SDA size as measured by the logarithm of expenditures 
(measures). Included also is a variable for single-SDA States and for SDAs in region 4. to control for regional dif- 
ferences in State policies. (The latter variable has a marginal impact on the significance level of Slate policies.) In 
the models for welfare recipients only, the maximum AFDC payment for a family of four is included as a control 
variable. 
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Table 12 



Estimated Impacts of Local Characteristics Used in JTPA Allocation Formula on 
Percentage of JTPA Terminees with Various Characteristics 

(Standard Errors in Parentheses) 



Adults 


WelfaK- 
Recipients 


Dropouts 




Minorities 


Females 


Other Barriers to 
Employment 


55 Years 
and Older 


Unemployment rate 
(PY86) 


-0.21 
(0.23) 


-0.36** 
(0.15) 




-0.69*** 
(0!23) 


-0.21 
(0.19) 


-0.64*** 
(0.22) 


0.00 
(0.08) 


Percentage of families with incomes 
below poverty level (1980) 


0.20 
(0.18) 


0.17 
(0.12) 




0.17 
(0.18) 


0.16 
(0.16) 


-0.54*** 
(0.18) 


-0.18** 
(0.06) 




Youth 


Welfare 
Recipients 


Dropouts 


In School 


Minorities 


Females 


Other Barriers to 
Employment 


Ages 
18to21 


Unemployment rate 
(PY86) 


0.16 
(0.21) 


-0.64** 
(0.27) 


-0.05 
(0.41) 


-1.12*** 
(0.53) 


0.04 
(0.15) 


-0.21 
(0.30) 


0.37 
(0.36) 


Percentage of families with incomes 
below poverty level (1980) 


-0.10 
(0.17) 


0.08 
(0.21) 


0.20 
(0.33) 


-0.27 
(0.21) 


0.00 
(0.12) 


-0.42* 
(0.24) 


-0.64** 
(0.29) 



* Statistically significant at ,10 level 
** Statistically significant at ,05 level. 
*** Statistically significant at .01 level. 



Quantitative Aiiaiysis 



programs,* and the last section of this chap- 
ter examines the impact of performance 
standards on whether SDAs have adopted 
specific intake procedures or service- 
provider arrangements. The relationships be- 
tween performance standards and SDA 
management practices are also explored in 
depth in the qualitative section. 

Results for Adults 

Table 13 presents the estimated impact of 
enroltaient criteria on the types of adults 
served in JTPA programs. The director ques- 
toxti aire asked directors which of several en- 
rollment criteria were used to select appli- 
caiits for enroltaient into each of four types 
of service: basic educational skills training, 
classroom training in occupational skills, 
OJT, and job-search assistance (including 
pre-employment work maturity training for 
youth). From these answers, we calculated 
the proportion of services for which each 
criterion was used. 

The first criterion is either to take 
everyone who was eligible or to select on a 
first-come, first-served basis but not to estab- 
lish any other criteria. These criteria were 
used on average for 18% of the activities. 
Having no specific criteria for excluding 
clients does not significantly influence the 
types of adults enrolled in JTPA programs, 
contrary to our initial hypothesis. As dis- 
cussed in the qualitative sections, targeting 
hard-to-serve clients often requires specific 
actions by the SDA or providers. 

The second criterion is a combination of 
either requiring a high school degree or GED 
for enrollment in an activity or using tests of 
basic skills attainment, either using low or 
high skills as a requirement.** These criteria 



are used on average for over 40% of the ac- 
tivities. Surprisingly, using these formal 
criteria significantly increases the percentage 
of welfare recipients and minorities enrolled 
in JTPA programs, and is even estimated to 
have a positive effect on the percentage of 
dropouts enrolled. Our case studies indicated 
that often these criteria were used to slot ap- 
plicants into the appropriate activity, so that 
individuals who have a high school diploma, 
for example, were given occupational skills 
training while those without a diploma were 
given other tra-nini;, such as basic skills 
remediation, iiimilarly, if priority for some 
activity was given to those with high basic 
skills, those with low skills were not neces- 
sarily turned away from JTPA programs but 
instead were often assigned to some other ac- 
tivity. These criteria, therefore, are not neces- 
sarily indicators that SDAs are enrolling only 
those with high attainment but rather that 
they are using objective criteria to slot in- 
dividuals into appropriate activities. 

In contrast, the next criterion, selecting 
those who are judged more likely to com- 
plete the program, significantly reduces the 
percentage of dropouts and older adults 
served and is estimated to have a negative im- 
pact on all the hard-to-serve groups. Thus, 
using subjective judgment about applicants is 
significantly more likely to reduce the per- 
centage of hard-to-serve clients in JTPA 
programs than using objective criteria, even 
if the latter are used to exclude individuals 
from certain activities. (Subjective criteria 
were used for an average of 36% of the adult 
activities and 31% of the youth activities.) 

SDAs that use previous work history as 
an enrollment criterion serve significantly 
fewer dropouts than other SDas.*** Finally, 



*The impacts of these procedures are esUmated controlling for the Stale policies, PIC and LEO concerns and in 
fluencc, and characteristics of the eligible population and local area. 

♦♦The criteria of educational attainment and the use of basic skills tests were combined bajed on preliminary 
factor analysis that found these criteria were correlated and prcUminary regression analysis that found these criteria 
had similar effects on clients. 

♦♦♦Previous work history is used as a c iwiluu tor i8% of adult activities and 28% of youth activities. 
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f Table 13 

f Estimated Impacts of Enrollment Criteria on Percentage of JTPA Terminees 

with Various Characteristics — Adults 



(Standard Errors in Parentheses) 



|i Welfare Dropouts Minorities Females Other Barriers to 55 Years and 

Recipients Employment Older 



Proportion of program activities 
for which criteria used: 



No specific criteria 


-0.46 
(1.72) 


-0.17 
(1.18) 


Educational or basic 
skills criteria 


3.61** 
(1.63) 


1.17 
(1.12) 


Judged more likely 
to complete program 


-1.77 
(1.26) 


-1.75** 
(0.87) 


Previous work history 


1.55 
(2.27) 


-3.12** 
(1.56) 


Referred liom other ^gtucy 


2.95** 
(1.41) 


0.57 
(0.96) 


Percentage of OJT participants 
selected first by employer 


-0.05* 
(0.03) 


-0.01 
(0.02) 


SDA established procedures 
to recruit and enroll 
speciflc group 


-1.03 
(1.02) 


0.77 
(0.60) 



-0'.31 




-1.21 


-0.04 










4.32*** 


1.31 


-0.60 


-0.70 


(1.69) 


(1.44> 


(1.68) 


(0.60) 


-0.91 


-1.00 


-1.19 


-1.11** 


(1.31) 


(1.12) 


(1.30) 


(0.46) 


-1.46 


1.52 


-1.52 


0.05 


(2.34) 


(2.00) 


(2.33) 


(0.82) 


1.57 


0.30 


-0.98 


0.10 


(1.45) 


(1.24) 


(1.46) 


(0.51) 


-0.03 


-0.03 


-0.04 


0.01 


(0.03) 


(0.02) 


(0.03) 


(0.01) 


0.67 




2.84*** 




(0.91) 




(0.99) 





* Statistically significant at JO level 

* Statistically significant at .05 level, 

* Statistically significant at .01 level. 



Quantitative Analysis 



SDAs that use referrals from other agencies 
serve significantly more welfare recipients, 
reflecting that many SDAs have formal and 
informal l?iiks with the welfare system in 
their jurisdictions.* 

In some SDAs, employers who are 
familiar with the eligibility rcquircments of 
JTPA programs will send their selected ap- 
plicants who are eligible for JTPA services to 
the SDA to get OJT subsidy for that person. 
The director questionnaire asked what 
proportion of OJT participants were first 
selected by employers and then referred to 
JTPA programs. Approximately 16% of the 
OJT participants are selected first by the 
employer. The results indicate that this prac- 
tice of accepting "reverse referrals" sig- 
nificantly reduces the percentage of adult 
welfare recipients serx^ed by JTPA programs 
and tends to have negative impacts on other 
hard-to-serve groups as well. 

The directors were also asked if the SDA 
had established procedures to recruit and en- 
roll specific client groups. Although the ef- 
fect is not statistically significant, SDAs that 
had established such procedures for welfare 
recipients actually served fewer welfare 
clients. This unexpected result may be due to 
reverse causation, where SDAs that were 
having trouble enrolling enough welfare 
recipients were more likely to establish proce- 
dures to correct the problem. Establishing 
specific procedures for those with other bar- 
riers to employment is estimated to have a 
large and significant impact, increasing the 
percentage with other barriers by almost 3 
percentage points. 

Results for Youth 

Table 14 presents the estimated impact of 
different enrollment criteria for youth. The 
questionnaire did not distinguish enrollment 
criteria for in-school and Tor out-of-school 
programs and to some extent makes the 



results less interpretable for youth than for 
adults. Nonetheless, the results represent the 
overall impact of these criteria on the types 
of youth served in JTPA programs. 

SDAs that do not establish any specific 
criteria serve significantly fewer in-school 
youth and youth with other barriers to 
employment and serve significantly more 
youth ages 18 to 21. Thus, simply tak.ing all 
eligible clients or selecting clients on a first- 
come, first-served basis results in serving 
youth who are likely to enroll m employment- 
oriented programs. 

SDAs that use formal education or ba?ic 
skills criteria serve a significantly greater per- 
centage of several hard-to-serve groups, in- 
cluding welfare recipients, minorities, and 
women. This pattem is similar to that ob- 
served for adults. 

Furthermore, as was also the case for 
adults, using subjective judgment about 
whether youth are likely to complete the 
program reduces service to the haid to serve, 
significantly so for welfare recipients, 
minorities, and those with other barriers. 
Again, use of subjective judgment to select 
individuals for JTPA programs is more a bar- 
rier for hard-to-serve groups than use of ob- 
jective criteria. 

Although a criterion of previous work his- 
tory significantly reduces service to dropouts 
for adults, it increases service to dropouts for 
youth, probably because young dropouts are 
more likely to have been employed than 
other youth. The use of referrals from other 
agencies significantly increases the percent- 
age of youth with other barriers to employ- 
ment served by JTPA programs. 

The practice of using reverse referrals 
from employers for OJT also reduces service 
to youth liard-to-serve groups, significantly 
so for welfare recipients, minorities, and 
women. 



*Refen:als from other agencies are used for 18% of adult activities and 26% of youth activities. 
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Table 14 

Estimated Impacts of Enrollment Criteria on Percentage of JTPA Terminees 
with Various Characteristics — Youth 

(Standard Errors in Parentheses) 



Welfare Dropouts In School Minorities Females Other Barriers to Ages 
Recipients Employment 1 8 to 21 



Proportion of program activities 
for which criteria used: 



No specific criteria 


0.96 


2.61 


-5.76* 


-2.43 


1.00 


-4.33'' 


6.51 


(1.68) 


(2.06) 


(3.22) 


(2.04) 


(1.20) 


(2.41) 


(2.87) 


Educational or basic 


4.83*** 


-0.72 


-0.09 


3.17* 


1.87* 


-3.56 


-0.45 


skills criteria 


(1.47) 


(1.80) 


(2.82) 


(1.79) 


(1.05) 


(2.11) 


(2.50) 


Judged more likely 


-2.49* 


0.65 


-2.41 


-3.32** 


-0.26 


-4.73** 


-0.05 


to complete program 


(1.28) 


(1.57) 


(2.47) 


(1.57) 


(0.92) 


(1.84) 


(2.19) 


Previous work history 


-2.61 


3.34* 


-1.81 


-1.09 


-0.22 


0.45 


2.13 


(1.57) 


(1.52) 


(3.00) 


(1.91) 


(1.12) 


(2.24) 


(2.66) 


Referred from other agency 


1.50 


-2.22 


3.53 


1.88 


-0.76 


3.89* 


0.53 




(1.39) 


(1.69) 


(2.65) 


(1.69) 


(0.99) 


(1.99) 


(2.35) 


Percentage of OJT participants 


-0.06** 


-0.01 


-0.04 


-0.06* 


-0.03* 


0.01 


0.07 


selected first by employer 


(0.03) 


(0.03) 


(0.05) 


(0.03) 


(0.02) 


(0.04) 


(0.05) 


SDA established procedures 
















to recruit and enroll: 
















Specific group 


-0.52 






0.79 




3.75*** 






(0.87) 






(1.06) 




(1.33) 




Dropouts 




g 14*** 


-8 28*** 








6.04* 






(L27) 


(1.98) 








(1.77) 


In School 




-4 42*** 


8.38*** 








-2.94* 






(M7) 


(1.82) 








(1.69) 


Ages 14 to 15 














-5.09* 



(1-67) 

* Statistically significant at .10 level 

* Statistically significant at .05 level. 

* .Statistically significant at .01 level. o 5 
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The directors were also asked about pro- 
cedures to recruit and enroll specific youth 
client groups. These procedures were effec- 
tive in significantly increasing tlie percentage 
of dropouts, in-school youth, and youth with 
other barriers to employment when these 
groups were targeted. Table 14 also presents 
the impact of targeting dropouts, in-school 
youth, and those under 16 for related client 
groups. These results indicate a clear tradeoff 
between targeting dropouts and targeting in- 
school youth. Furthermore, targeting 
dropouts will also significantly increase the 
percentage of youth ages 18 to 21. 

Summary of Impact 

of Enrollment Criteria 

The criteria tliat SDAs use to select par- 
ticipants have important impacts on the 
proportion of hard-to-serve clients participat- 
ing in JTPA programs. The use of education- 
al and basic skills criteria are associated with 
greater swvice to hard-to-serve groups, 
primarily because SDAs tend to use these 
criteria to slot individuals into appropriate 
services, as is discussed further in our qualita- 
tive analyses. In contrast, SDAs that rely on 
subjective judgment about individuals' 
likelihood of completing the program consis- 
tently s«ve fewer hard-to-serve groups for 
adults and youth. Furthermore, SDAs that 
allow employers to make a substantial 
amount of "reverse referrals" to their OJT 
program also serve significantly fewer hard- 
to-serve individuals. 

Impact of Service-Provider 
Arrangements 

SDAs were asked about the types of ser- 



vice providers that provided four types of 
program activities.* Our analysis examined 
the impact of the proportion of the four types 
of activities that were provided by various 
providers. We also estimated the impacts of 
the percentage of expenditure in perfor- 
mance-based contracts and, for adults, the 
terms of those contracts.** Table 15 presents 
these results for adults. 

Results for Adults 

The proportion of activities provided by 
the SDA itself is significantly associated 
with lower service to welfare recipients. 
This result is unexpected; our initial 
hypothesis was that SDAs that retained 
responsibility for provision of services would 
serve more hard-to-serve clients. Use of the 
public schools, including community col- 
leges, also tends to reduce the percentage of 
hard-to-serve groups served, although none 
of the effects are statistically significant. Use 
of the employment service appears to sig- 
nificantly increase the percentage of older 
adults served by JTPA programs but has no 
significant effect on the other types of clients 
enrolled in the program. 

The proportion of program activi^ 
provided by CBOs significantly increases ser- 
vice to dropouts and those with other barriers 
to employment and has a positive, although 
not statistically significant, impact on 
minorities and females. Because many CBOs 
are organized to serve specific target groups, 
using them to provide JTPA services appears 
to be an effective way to increase service to 
those groups, as confirmed by our qualitative 
analysis. In contrast, use of for-profit pro- 
viders is associated with a lower percentage 



♦Originally we tried to ask about the amount of funding for various types of service providers, but in the pretest 
we found that many SDAs could not provide that information. 

**The impacts of service-provider arrangements are estimated while controlling for State policies. PIC and 
LEO concerns and influence, SDA enrollment criteria, and characteristics of the eligible population and the local 
area. Because the infoimation about types of service providers and percentage of expenditures in performance- 
based contracts was collected in the fiscal/MIS questionnaire, this analysis is coi ducted on the sample of SDAs that 
responded to both the questionnaires. This sample is somewhat smaller than the sample used for the above 
analyses, which were based on the sample that responded to the director questionnaire. 
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Table 15 

Estimated Impact of Service-Provider Arrangements on Percentage of 
JTPA Participants with Various Characteristics— Adults 

(Standard Errors in Parentheses) 



Welfare Dropouts Minorities Females Other Barriers to 55 Years and 

Recipients Employment Older 



Proportion of program 
activities provided by 



SDA itself 


-3.17* 


-0.20 


-1.66 


0.09 


0.95 


-0.54 




(1.75) 


(1.17) 


(1.82) 


(1.53) 


(1.72) 


(0.73) 


Public schools (including 


-3.32 


-1.46 


-2.99 


0.66 


-0.86 


0.91 


conmiunity college) 


(2.52) 


(1.69) 


(2.67) 


(2.20) 


(2.50) 


(1.04) 


Employment services and 


-0.74 


-0.89 


-A. 00 


0.70 


1.78 


1.46* 


other Government agencies 


(2.00) 


(1.35) 


(2.14) 


(1.77) 


(1.97) 


(0.85) 


Gonmiunity-based 


-1.02 


2.55** 


1.97 


0.63 


3.17* 


-0.01 


organizations 


(1.66) 


(1.10) 


(1.72) 


(1.45) 


(1.62) 


(0.68) 


For-profit organizations 


2.19 


0.10 


-2.69 


1.97 


-3.58* 


-0.18 




(1.45) 


(1.31) 


(2.03) 


(1.72) 


(1.92) 


(0.81) 


Percentage of adult expenditures 


-0.011 


-0.014 


0.039* 


0.025 


-0.007 


0.001 


in performance-based contracts 


(0.021) 


(0.014) 


(0.021) 


(0.018) 


(0.020) 


(0.009) 


Terms of performance-based 














contracts (weighted by percentage 












of expenditures in performance- 












based contracts) 














Wage rate 


-0.046*** 


-0.013 


-0.009 


-0.017 


-0.005 


0.005 




(0.014) 


(0.009) 


(0.014) 


(0.012) 


(0.013) 


(0.006) 


Proportion of full payment 0.008 


-0.001 


0.029 


0.046 


-0.081 


*0.022 


for placement outcomes 


(0.047) 


(0.031) 


(0.049) 


(0.041) 


(0.046) 


(0.019) 


Whether terms of 


0.100* 


0.103*** 


-0.060 


0.109** 


-0.044 


-0.033 


performance-based 


(0.059) 


(0.039) 


(0.061) 


(0.051) 


(0.058) 


(0.024) 


c(»itracts vary 









* Statistically significant at JO level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 
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of individuals with other barriers served by 
JTPA programs. 

SDAs were also asked the amount of ex- 
penditures that were in performance-based 
contracts. On average, SDAs spend ap- 
proximately 25% of their total expenditures 
in performance-based contracts. As discussed 
in our conceptual framework, it is often ar- 
gued that performance-based contracts in- 
crease providers' incentives to enroll the 
most job ready and, therefore, reduce service 
to hard-to-serve groups. The results in Table 
15, however, do not support that contention. 
In fact, the percentage of adult expenditures 
in performance-based contracts is associated 
with significandy greater service to 
minorities. 

The directors were further asked about 
the terms of their performance-based con- 
tracts (about their largest contract if the terms 
varied). In the analysis, we weighted the 
answers about the contract terms by the per- 
centage of expenditures in such contracts.* 
These results indicate that high wage-rate re- 
quirements in performance-based contracts 
tend to reduce service to hard-to-serve 
groups, significandy so for welfare recip- 
ients. The proportion of the full payment that 
is reserved for placement outcomes (includ- 
ing retention) significantiy reduces service to 
those with other barriers to employment but 
not to other groups. The lack of consistent 
results for tiiis important dimension may be 
because there are a variety of ways that 
SDAs ensure that providers meet the terms of 
their contract besides holding back a large 
proportion of final payment, as discussed in 
the qualitative analysis. 

SDAs that vary the terms of their con- 
tracts strve significantiy more welfare 



recipients, dropouts, and females than do 
SDAs that impose the same terms on all con- 
tractors. In our qualitative analysis we 
describe how SDAs in our site visit sample 
varied these terms. 

Results for Youth 

Table 16 presents the estimated impacts 
of service-provider arrangements on youth. 
Because we did not distinguish secondary 
and post-secondary schools, some of the 
results reflect SDA decisions about serving 
in-school and out-of-school youth. As ex- 
pected, the use of public schools as service 
providers is associated with a significant 
reduction in the percentage of dropouts that 
receive JTPA services. Provision of services 
by the SDA itself is significantly associated 
with a lower percentage of in-school youth. 
Reliance on the employirent service as a ser- 
vice provider also is associated with fewer 
in-school youth as well as fewer minorities 
served. 

The pattern of CBOs serving more hard- 
to-serve groups tiiat was observed for adults 
also exists for youth, but the results are sig- 
nificant only for dropouts. Similarly, the 
proportion of activities provided by for-profit 
organizations is associated with lower ser- 
vice to in-school youth, minorities, and those 
with other barriers to employment, although 
none of these effects are statistically sig- 
nificant. For-profit providers, however, are 
associated with increased service to women, 
an effect that is significant for youth. 

As was observed for adults, the percent- 
age of youth expenditures in performance- 
based contracts did not reduce service to 
hard-to-serve groups and, in fact, is as- 
sociated witli a small increase in the percent- 
age of youth with other barriers to 



♦The impacts of the percentage of expenditures in performance-based contracts are estimated in a separate 
model without the variables for the terms of the contrscts and therefore represent the impacts of perfonnance-based 
contracts with average terms. 
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Table 16 

Estimated Impacts of Service*Provider Arrangements on Percentage of 
JTPA Participants with Various Characteristics— Youth 

(Standard Errors in Parentheses) 





Welfare 
Recipients 


Dropouts 


In School 


Minorities 


Females 


Other Barriers to 
Employment 


Ages 
18 to 21 


Proportion of program 
activities provided by 
















SDA itself 


-0.27 
(1.67) 


2.49 
(2.02) 


-5.57* 
(2.98) 


-1.12 
(2.00) 


1.54 
(1.15) 


-1.33 
(2.26) 


2.02 
(2.81) 


Public schools 

(including community college) 


-0.45 
(2.42) 


-5.21 
(2.96) 


*3.29 
(4.61) 


-t.67 
(2.92) 


1.07 
(1.66) 


1.88 
(3.30) 


4.41 
(4.13) 


Employment services and 
other Government agencies 


-2.77 
(1.97) 


1.67 
(2.40) 


-6.78* 
(3.75) 


-5.04** 
(2.38) 


1.64 
(1.35) 


-0.89 
(2.67) 


3.60 
(3.34) 


Conununity-based 
organizations 


0.37 
(1.62) 


3.84* 
(1.96) 


-0.36 
(3.05) 


0.74 
(1.95) 


0.48 
(1.12) 


1.12 
(2.18) 


-1.10 
(2.73) 


For-profit 
oiganizations 


0.53 
(1.90) 


0.87 
(2.32) 


-1.75 
(3.60) 


-1.75 
(2.27) 


3.02** 
(1.31) 


-4.06 
(2.57) 


-0.07 
(3.22) 


Percentage of youth expenditures 
in performance-based contracts 


-0.03 
(0.02) 


0.02 
(0.02) 


-0.03 
(0.04) 


0.02 
(0.02) 


0.02 
(0.01) 


0.04 
(0.03) 


-0.06* 
(0.03) 



* Statistically significant at JO level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level.- 
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employment and a decrease in the percentage 
of older youth served in JTPA programs.* 

Summary of Impact of 
Service-Provider Arrangements 
Service-provider arrangements have 
some impacts on the type of clients served in 
JTPA programs, although there were fewer 
effects than expected The use of CBOs as 
providers is associated with significantly 
greater service to dropouts for both adults 
and youth, while there is a weak pattern 
showing that the use of public schools as 
providers is associated with less service to 
sevCTal hard-to-serve groups. 

Contrary to initial hypotheses, use of per- 
formance-based contracts does not reduce 
service to the hard to serve. There is some 
evidence that setting a high wage r»,te reqi^^re- 
ment in such contracts reduces . ice tr 
adult welfare recipients. SDAs that var 
terms of their performance-based contracts 
serve significantiy more welfare recipients, 
dropouts, and women than SDAs that impose 
thc> same terms on all their performance- 
based contracts. 

Impact of the Types 
of Program Services 

The next step is to examine the inter- 
relationships between the types of clients 
served and the types of services offered. As 
discussed above, there is a strong relation- 
ship between the types of services offered 
and the types of clients served, but the direc- 
tion of causation is not clear and may vary 
by SDA. Based on the Imdings from the site 
visit interviews, our best assessment of the 



relationship is that SDAs establish planned 
target groups that may require specific 
program services so that, in the planning 
stage, planned clients affect planned services. 
After these planning decisions are made, 
however, the types of services actually of- 
fered affect the relative numbers of clients 
with different characteristics that are enrolled 
in JTPA programs. Thus, we have included 
the proportion of services in different 
program activities as predictors of the types 
of clients served** In the next chapter we ex- 
amine the impact of performance standards 
and other policies and practices on the choice 
of program activities. 

Results for Adults 

The SDAs reported the number of in- 
dividuals receiving various types of program 
activities.*** Table 17 presents the estimated 
impact of the percentage of services that 
were in various program activities for adults. 
The reference group is the percentage of ac- 
tivities in classroom training in occupational 
skills. 

SDAs that place greater emphasis on job- 
search assistance than classroom training 
serve significantiy fewer welfare recipients 
and women. These SDAs, however, serve sig- 
nificantiy more dropouts. There arc two pos- 
sible reasons for the latter rclationship. First, 
dropouts are possibly excluded from class- 
room training in some SDAs (particularly 
when educational institutions are used to 
provide classroom training) and arc given 
job-search assistance instead. Alternatively, 
SDAs may make room for the move intensive 
needs of dropouts by providing less expen- 



* Although we examined the effects of the temis of performance-based contracts for youth, we were unable to 
detect ccMisistait and significant effects. Thus, the terms of performance-based contracts were not included in the 
youth model. 

**The effects of program activities are estimated whUc controlling for State policies, PIC and LEO concerns 
and influence, SDA enrollment criteria, service-provider arrangements, and characteristics of the eligible peculation 
and local area The sample included SDAs that responded to both the director and fiscal/MIS questionnaires. 

***SDAs reported these figures in a variety of ways. Thice-fourths reported the number of individuals that had 
ever received an activity so that those who received more than one activity were counted more tban once. We calcu- 
lated the proportion of services received by individuals that wctc in the differ^t program activity categories. 
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Table 17 

Estimated Impact of Program Activities and Supportive Services on Percentage of 
JTPA Participants with Various Characteristics— Adults 

(Standard Errors in Parentheses) 



Welfare Dropouts Minorities Females Other Barriers to 55 Years and 

Recipients Employment Older 



Percentage cf program activities 
(relative to classroom training in 
occupational skills): 



Job-search assistance 


-0.08** 
(0.03) 


0.07*** 
(0.02) 


-0.05 
(0.03) 


-0.09*** 
(0.03) 


-0.00 
(0.03) 


-0.01 
(0.01) 


Basic skills 


0.03 
(0.06) 


0.22*** 
(0.04) 


0.16*** 
(0.06) 


0.02 
(0.05) 


0.13** 
(0.06) 


-0.00 
(0.03) 


OJT 


-0.14*** 
(0.04) 


0.03 
(0.03) 


-0.10** 
(0.04) 


-0.17*** 
(0.03) 


-0.08** 
(0.04) 


-0.02 
(0.02) 


Wo± experience 


0.17** 
(0.08) 


0.01 
(0.05) 


0.04 
(0.08) 


0.12* 
(0.06) 


0.04 
(0.08) 


-0.01 
(0.03) 


Percentage of expenditures 
in supportive services 


0.02 
(0.11) 


-0.03 
(0.07) 


0.21* 
(0.11) 


-0.02 
(0.09) 


-0.00 
(0.11) 


-0.01 
(0.05) 



* Statistically significant at .10 level. 

* Statistically significant at .05 level. 

* Statistically significant at .01 level. 
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sive services to other groups. In that case, 
SDAs that offer more job-search assistance 
are able to serve more dropouts for the same 
average cost. Both of these relationships 
v/trt found in our case studies. 

SDAs that provide more basic skills 
remediation relative to classroom training in 
occupational skills enroll significantly more 
dropouts, minorities, and those with other 
barriers to employment. In contrast, SDAs 
that emphasize OJT relative to classroom 
training enroll significantly fewer welfare 
recipients, minorities, women, or those with 
other barriers to employment. The JTPA 
legislation restricts work experience in the 
program. SDAs that provide work experience 
serve significantly more welfare recipients 
and women. 

The JTPA legislation also restricts expen- 
ditures on supportive services, and the lack 
of such services has frequently been cited as 
a barrier to enrolling the hard to serve. The 
percentage of expenditures in supportive ser- 
vices, however, is not generally associated 
with service to different types of clients, al- 
though it is associated with a modest increase 
in the percentage of minorities served.* 

Results for Youth 

Table 18 presents the impact of program 
activities on the characteristics of youth 
served in JTPA programs. For youth, the 
choice between whom to serve and the types 
of services to offer is probably more simul- 
taneous than for adults because of the SDA*s 
decision of whether to offer in-school 
programs. Nonetheless, these results indicate 
that a strong relationship exists between the 
types of services and the types of clients for 
youth. The provision of more job-search as- 
sistance, including pre-employment/work 
maturity training, is significantiy associated 
with serving more in-school youth and fewer 



older youth and welfare recipients. Providing 
basic skills training has a large and signifi- 
cant impact on the percentage of dropouts 
served. Basic skills training, however, is as- 
sociated with reduced service to welfare 
recipients for youth as well as reduced ser- 
vice to older youth. SDAs that emphasize 
OJT also tend to serve fewer welfare 
recipients and women, similar to effects ob- 
served for adults. Both work experience and 
tryout employment are associated with in- 
creased service to in-school youth and 
reduced service to older youth. 

The percentage of expenditures in suppor- 
tive services significantly increases the per- 
centage of youth who are minorities and 
female, although there is an unexpectedly 
negative association with the percentage of 
youth with other barriers to employment 

Summary of Impact of Types 

of Program Services 

These results indicate that the types of 
services offered do have strong associations 
with the types of clients served. Basic skills 
remediation has the greatest association with 
service to hard-to-serve groups. Classroom 
training in occupational training (the 
reference group) is next, followed by job- 
search assistance. SDAs that provide more 
OJT are the least likely to serve hard-to-serve 
groups. The next chapter examines in depth 
the factors that affect the SDAs* choices of 
the types of services offered in JTPA 
programs. 

SDA Practices as Mechanisms for 
Performance-Standard Effects 

In this chapter we have found that some 
aspects of State performance-standards and 
related policies affect the percentage of hard- 
to-serve clients served by JTPA programs. 
We have also found that several SDA prac- 



*Some supportive services are possibly provided through performance-based contracts so that the amount 
reported in the questionnaire understates the total support services received. 
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Table 18 



Estimated Impact of Program Activities and Supportive Services on Percentage of 
JTPA Participants with Various Characteristics — Youth 

(Standard Errors in Parentheses) 





Welrare 
Recipients 


Dropouts 


In School 


Minonties 


Females 


Other Barriers to 
Employment 


Ages 
18 to 21 


Percentage of program activities 
(relative to classroom training in 

OCClipUllOrlUt' SKlUSjm 
















Job-search assistance (including 
pre-employment/work maturity) 


-0.12*** 
(0.04) 


-0.04 
(0.05) 


0.22*** 
(0.07) 


0.00 
(0.05) 


-0.02 
(0.03) 


-0.02 
(0.05) 


-0.15** 
(0.07) 


Basic skills 


-0.13** 
(0.05) 


0.22*** 
(0.06) 


0.02 
(0.09) 


-0.02 
(0.06) 


0.01 
(0.04) 


0.03 
(0.07) 


-0.16** 
(0.09) 


OJT 


-0.13** 
(0.05) 


-0.04 
(0.06) 


-0.10 
(0.10) 


-0.10 
(0.07) 


-0.06* 
(0.04) 


-0.23 
(0.07) 


0.10 
(0.09) 


Work experience 


-0.07* 
(0.04) 


-0.06 
(0.05) 


0.18*** 
(0.07) 


-0.00 
(0 05) 


0.00 
(0.03) 


0.03 
(0.05) 


-0.09* 
(0.07) 


Tryout employment 


-0.07** 
(0.04) 


-0.05 
(0.04) 


0.25*** 
(0.07) 


0.01 
(0.04) 


-0.01 
(0.03; 


0.00 
(0.05) 


-0.15*** 
(0.06) 


Percentage of expenditures 
in supportive services 


0.05 
(0.11) 


-0.09 
(0.13) 


0.01 
(0.20) 


0.33** 
(0.13) 


0.24*** 
(0.07) 


-0.25* 
(0.15) 


0.22 
(0.18) 



* Statistically significant at .10 level. 

* Statistically significant at .05 level. 

* Statistically significant at .01 level. 
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rices, including enrollment criteria and ser- 
vice-provider arrangements, also affect 
clients. The quesrion remains whether State 
performance-standards policies have led 
SDAs to adopt these pracrices, that is, 
whether the SDA enrollment criteria and ser- 
vice-provider pracrices are the mechanisms 
through which performance-standards 
policies affect clients. 

To examine this issue, we have esrimated 
the impact of State policies on SDA enroll- 
ment criteria, policies toward specific target 
groups, types of service providers used, use 
of performance-based contracts, and terms of 
those contracts. We have limited the analysis 
to those pracrices that were found to have 
some impact on clients or that are of par- 
ricular policy interest. Furthermore, because 
these pracrices are less predictable than our 
main outcomes, we restricted the State 
policies to three primary variables: the use of 
the oprional DOL models, the emphasis on 
exceeding standards, and the weight on the 
cost standard. We also examined the impact 
of State policies for serving paiticular client 
groups on the SDA choice to target services 
to that particular group. 

SDA Enrollment Practices 

Table 19 presents the esrimated impacts 
of State performance-standard policies on 
SDA enrollment criteria and on whether 
SDAs have established procedures to recruit 
and enroll specific client groups for adults. 
SDAs in States that use the DOL models are 
significantly less likely to use two enrollment 
criteria that were found to reduce service to 
the hard to serve: subjecrive judgments of 
whether the individual is likely to complete 
the program and reverse referrals from 
employers for OJT slots. Furthermore, SDAs 
in States that use the adjustment models are 
also significantly more likely to establish pro- 
cedures to recruit and enroll welfare 
recipients and those with other barriers to 
employment. 

SDAs in States with policies that em- 



phasize exceeding standards are significantly 
less likely to use formal enrollment criteria 
of basic skills or educarional attainment, 
criteria found to increase service to hard-to- 
serve groups. State policies that give more 
weight to the cost-per-entered-employment 
standard significandy increase the use of 
reverse referrals for OJT and significandy 
reduce the use of procedures to bring welfare 
recipients into JTPA programs. 

States that establish policies to encourage 
service to welfare recipients significantly in- 
crease the probability that SDAs will estab- 
lish procedures to bring welfare recipients 
into the program. However, State policies for 
dropouts or for those with other barriers to 
employment are not translated into local prac- 
rices. 

The impacts of State policies on SDA en- 
rollment pracrices for youth, presented in 
Table 20, are very similar to the pattern ob- 
served for adults. The use of the DOL 
models significandy increases SDAs' target- 
ing of service to youth welfare recipients, 
dropouts, and those with other barriers to 
employment. State policies that emphasize 
exceeding performance standards significant- 
ly reduce the use of basic skills or education- 
al criteria, as for adults. State policies to 
encourage service to youth welfare recipients 
significantly increase the probability that 
SDAs will target services to that group, al- 
though the targering of services to dropouts 
and those with other barriers to employment 
appears to be primarily affected by local fac- 
tors for youth as well as adults. 

SDA Service-Provider Arrangements 

Table 21 presents the esrimated impacts 
of State performance-standard policies on 
SDA service-provider arrangements. The use 
of the model significandy increases the use 
of public schools as service providers but 
does not have any other impact on providers 
or contract provisions. State policies that em- 
phasize exceeding standards, however, have 
substantial effects on service-provider arran- 
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Table 19 



Estimated Impacts of State Policies on SDA Enrollment Criter ia and Procedures 
to Enroll and Recruit Speciflc Groups — Adults 



Enrollment Criteria 



Proportion of Activities SDA Procedures to Enroll and 

for Which Criteria Used Recruit Specific Groups 



State Policies 


Educational or 
Basic Skills 


Judged More 
Likely to Com- 
plete Program 


Percenta&e of 
OJT Participants 
Selected First 
by Employer 


Welfare 
Recipients 


Dropouts 


Other Barriers to 
Employment 


Use of model (number 
of standards for 
which model used) 


-0.01 
(0.01) 


-0.02* 
(0.01) 


-1.17** 
(0.49) 


0.05** 
(0.01) 


0.02 
(0.02) 


0.04** 
(0.01) 


Emphasis on exceeding 
standards in 6% 
policy 


-0.04** 
(0.02) 


0.02 
(0.02) 


-0.86 
(0.99) 


-0.02 
(0.03) 


0.02 
(0.03) 


-0.02 
(0.03) 


Incentive weight on 
cost-per-entered 
employment standard 


0.003 
-(0.003) 


-0.00 
(0.003) 


10.22* 
(0.12) 


-0.008* 
(0.005) 


-0.002 
(0.004) 


0.001 
(0.004) 


State policy for 
serving specific group 








0.13** 
(0.05) 


-0.01 
(0.06) 


-0.01 
(0.05) 



* Statistically significant at .10 level. 

* Statistically significant at .05 level. 

* Statistically significant at .01 level. 



Table 20 

Estimated Impacts of State Policies on SDA Enrollment Criteria and Procedures 
to Enroll and Recruit Specific Groups — Youth 



Enrollment Criteria 



Proportion of Activities 
for Which Criteria Used 



SDA Procedures to Enroll and 
Recruit Specific Groups 



State Policies 



Educational Judged More Likely Welfare Dropouts 
or Basic Skills to Complete Program Recipients 



Other Barriers to 
Employment 



Use of model (number of 
standards for which model used) 



-0.01 
(0.01) 



-0.01 
(0.01) 



0.03** 
(0.02) 



0.03** 
(0.01) 



0.03** 
(0.01) 



Emphasis on exceeding 
standards in 6% policy 



-0.05** 
(0.02) 



-0.01 
(0.02) 



-0.00 
(0.03) 



-0.0 i 
(C.03) 



-0.03 
(0.03) 



Incentive weight on cost-per- 
vTitered-employment standard 



0.003 
(0.003) 



0.003 
(0.003) 



-0.004 
(0.004) 



-0.002 
(0.004) 



0.002 
(0.004) 



State policy for serving specific group 



0.12** 
(0.05) 



0.01 
(0.05) 



-0.01 
(0.05) 



* Statistically significant at .10 level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 
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Table 21 

Estimated Impacts of State Policies on Servics-Provider Arrangements 



Proportion of 
Activities Provided by 



Public 
Schools 



CBOs 



For-Profit 
Organizations 



Percentage 
of Adult 
Expenditures 
in Performance- 
Based Contracts 



Terms of Adult 
Performance-Based Contracts 



Vary Wage Proportion of 

Terms Rate Payment for 

Placement Outcomes 



Use of model (number of 0.02** 0.01 -0.00 

standards for which model used) (0.01) (0.01) (o!oi) 



0.82 
(0.89) 



0.00 
(0.01) 



0.01 
(0.03) 



0.48 
(0.94) 



Emphasis on exceeding 
standards in 6% policy 



0.00 
(0.01) 



0.01 
(0.02) 



0.06*** 
(0.02) 



5.77*** 
(1.78) 



-0.01 
(0.02) 



0.10* 
(0.06) 



3.62* 
(2.05) 



Incentive weight on cost-per- -0.001 0.001 -0.002 

entered-employment standard (0.002) (0.003) (0.002) 



-0.179 
(0.223) 



0.002 
(0.003) 



-0.004 
(0.007) 



-0.231 
(0.239) 



* Statistically significant at .10 level. 

* Statistically significant at .05 level. 

* Statistically significant at .01 level. 




Quantitative Analysis 



gements. Such policies significantly increase 
the use of for-profit providers, which we 
found reduced service to some hard-to-serve 
groups. SDAs in States that emphasized ex- 
ceeding standards also expended significant- 
ly more funds in performance-based con- 
tracts for adults, and these policies also affect 
the terms of those contracts, increasing the 
wage rate required and the fraction of full 
payment reserved for placement outcomes. 
Although we found only small effects of 
these terms on clients served in the quantita- 
tive analysis, we will return to this issue in 
the qualitative analysis. The weight on the 
cost standard did not significantly affect 
SDAs' service-provider arrangements. 

Summary of SD A Practices 
as Mechanisms for 
Performance Standards 

These results indicate that enrollment 
criteria and contracting practices are 
mechanisms through which performance 
standards affect clients served. SDAs in 
States that use the adjustment models are 
more likely to target services to hard-to-serve 
groups and are less likely to use enrollment 
criteria that reduce service to hard-to-serve 
groups. SDAs in States that emphasize ex- 
ceeding standards in their 6% policies are 
more likely to use for-profit providers and 
write performance-based contracts with 
terms that emphasize performance. SDAs in 
States that place greater weight on the cost 
standard are more likely to use enrollment 
criteria that reduce service to the hard to 
serve and are less likely to target services to 
welfare recipients. These mechanisms, 
however, do not fully explain the impacts of 
State policies on clients. State policies also 
affect clients in part through their effects on 
service mix, as we discuss in the next chap- 
ter, and in part through their effects on other 
management practices, as we discuss in 
Chapter XI. 

Summary and Conclusions 



The types of clients served in JTPA 
programs are affected by both State perfor- 
mance-standards policies and SDA practices. 
Several aspects of State performance-stand- 
ards policies can affect the extent to which 
hard-to-serve groups are served by JTPA 
programs: 

• Adjustment policies — ^Use of t*'^^* 
DOL adjustment models significantly 
increases service to several hard-to- 
serve groups, including adult welfare 
recipients and dropouts. Specifying 
procedures for adjustments beyond 
the model also tends to increase ser- 
vice to some hard-to-serve groups, 
but the effects are less consistent. 

• Emphasis on exceeding standards — 
States that place a strong emphasis on 
exceeding standards in their 6% 
policies tend to lead SDAs to enroll 
significantly fewer hard-to-serve 
clients and to focus their youth 
programs on older youth. 

• Emphasis on the cost standards — 
States that place a high weight on the 
cost standard lead SDAs to serve 
fewer welfare recipients and dropouts. 

• Use of 6% for hard to serve — ^These 
policies tend to increase service to 
welfare recipients and adults with 
other barriers to employment and to 
focus the youth programs on in- 
school youth. 

• Hard-to-serve policies — States that es- 
tablish policies for welfare recipients 
and dropouts, including State stand- 
ards for these groups, lead SDAs to 
enroll more of these hard-to-serve 
groups. 

SDA policies and practices also affect the 
types of clients served by JTPA programs: 

• PIC influence— PIC influence per se 
does not reduce service to the hard to 
serve. The results suggest that PICs 



105 



72 



Impact of Policies and Practices on Clients 



design of the program are associated 
with greater enrollment of the hard to 
serve while PICs that are more in- 
volved in contracting are associated 
with less service to some hard-to- 
serve groups. 

Enrollment criteria — SDAs that use 
objective basic skills or educational 
attainment criteria serve more hard-to- 
serve clients, probably because these 
criteria are used to slot participants 
into appropriate activities. In contrast, 
SDAs that use subjective judgments, 
require previous work histories, or ac- 
cept reverse referrals from employers 
for OJT slots serve significantly 
fewer hard-to-serve clients. 

Service-provider arrangements — ^The 
use of CBOs as service providers is 
associated with greater service to 
several hard-to-serve groups, includ- 
ing both adult and youth dropouts. 
The percentage of expenditures in per- 
formance-based contracts does not 
reduce service to hard-to-serve 
groups, although tiiere is a weak pat- 
tern showing that setting high wage- 
rate requirements or holding more 
payment back for placement out- 
comes reduces the percentage of 
some hard-to-serve groups in JTPA 
programs. SDAs that vary the terms 
of their contracts serve significantiy 
more adult welfare recipients and 
dropouts. 

Program services — The types of 
program services offered by SDAs 
have a strong influence on the types 
of clients enrolled in JTPA programs. 
Basic skills remediation has the 
strongest association with enrollment 
of hard-to-serve groups, followed by 
classroom training in occupational 



Based on 75lh and 25lh percentiles of the State polici^ 



skills, then job-search assistance. 
SDAs that provide more OJT tend to 
serve significantly fewer hard-to- 
serve clients. 
This chapter also examined whether the 
SDA practices are mechanisms through 
which performance standards affect clients. 
The results indicate that State performance- 
standards policies do significantly affect 
SDAs' enrollment criteria and contracting 
practices, which in part explains the effects 
of performance-standards policies on clients. 

The results of this chapter indicate that 
performance standards can influence the 
types of clients enrolled in JTPA programs. 
Altiiough the effects are statistically sig- 
nificant and have important policy implica- 
tions, it should be kept in mind that the 
effects are not large. Even in States with 
policies that are found to discourage service 
to hard-to-serve groups, SDAs are enrolling a 
considerable number of hard-to- serve clients. 

For example, on average, 29.5% of adult 
clients in JTPA programs are welfare 
recipients, approximately equal to their in- 
cidence in the eligible population, which is 
30.6%. The resuhs of this evaluation indicate 
that an SDA that is average on all other 
characteristics but is in a State that has a 
strong emphasis on exceeding standard? is es- 
timated to serve 28.3% welfare recipients; a 
similar SDA in a State with a low emphasis 
on exceeding is estimated to serve 30.2% wel- 
fare recipients.* An average SDA in a State 
that places a high weight on the cost standard 
is estimated to serve 26.9% welfare 
recipients; a similar SDA in a State tiiat 
places a low weight on the cost standard is es- 
timated to serve 31.2% welfare recipients. 

These examples indicate that SDAs in 
States with policies that discourage service to 
welfare recipients are serving welfare 
recipients only slightly less than their in- 
cidence in the population. Thus, the resuhs 
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Quantitative Analysis 



should not be interpreted that certain perfor- 
mance-standards policies preclude service to 



the hard to serve; rather the results indicate 
that some policies reduce service to the hard 
to serve at the margin. 



VL Impact of Policies and Practice on 
Services Provided and Program Costs 
in JTPA Programs: Evidence from 
the Quantitative Analysis 



Introduction 

In this chapter we examine the effects of 
State performance-standards policies and 
SDA practices on the types of services of- 
fered in JTPA programs and on program 
costs. It is important to determine the factors 
that affect the provision of services both be- 
cause of a direct concern about the quality 
and intensity of training and because the 
types of services offered have important ef- 
fects on the types of clients enrolled in the 
programs, as we demonstrated in the pre- 
vious chapter. It is also important to deter- 
mine how performance standards, which are 
intended to increase the cost-effectiveness 
of the nPA program, in fact affect program 
costs. 



Overview of Chapter 
and Key Findings 

We first present an analysis of program 
services . The structure of the analysis follows 
the conceptual framework in Chapter II. 
Thus, we begin by examining the impact of 
factors that are hypothesized to affect SDA 
design decisions, including 

• State performance-standards policies . 
We fmd that adjustment policies and 
policies for the hard to serve tend to 
increase the intensity of program ser- 
vices, while policies that emphasize 
exceeding standards reduce the 
amount of basic skills training 
provided and increase the employ- 
ment focus of JTPA training. 
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• PIC and LEO concerns . We find pat- 
terns consistent with those seen for 
clients whereby PICs influential in 
program design are associated with 
more intensive services while PICs 
more influential in contracting are 
associated with more OJT services. 

Next, we add SDA practices that may af- 
fect the types of services provided, 
specifically 

• SDA client goals . We find that SDAs 
that target specific client groups also 
provide program services appropriate 
for the needs of those clients. 

We then turn to an analysis of program 
costs. We start by examining the impact of 
factors hypothesized to affect program 
design, including 

• State performance-standards policies . 
We find that State performance-stand- 
ards policies have fewer effects on 
program costs than on clients or ser- 
vices. For example, the weight placed 
on the cost standard does not sig- 
nificantly reduce program costs. 

• PIC and LEO concerns and influence . 
Again, we find few significant effects 
on costs. 

Next, we examine whether SDA prac- 
tices affec^t program costs, including 

• Ser/ice-provider arrangements . We 
find some evidence that differences in 
ccsts among SDAs may reflect the ex- 
tent to which JTPA funds are lever- 
aged with other resources. 

Finally, we examine the interrelationship 
among program outcomes by examining the 
impact on program costs of the following: 

• Provider services . We find significant 
differences in the costs of program 
services, with job-search assistance 
being considerably less expensive 
than other services. 



• Clients served . Controlling for the 
types of services provided, types of 
clients served have only a small im- 
pact on program costs. 

Program Services 

Information about the number of in- 
dividuals receiving different types of 
program activities was obtained in our sur- 
vey of SDAs. Because SDAs are not required 
to report program activities to DOL, SDAs 
track the number of individuals receiving ser- 
vices in a variety of ways. The majority 
record the number of individuals who ever 
received a separate activity (so that those 
who received more than one program service 
were counted more than once), while other 
SDAs record only one activity per partici- 
pant. From the information reported to us, 
we calculated the percentage of the total 
number of participant services that were in 
each program activity. 

Table 22 presents program activity 
results for SDAs reporting only one activity 
per participant and for SDAs reporting multi- 
ple activities for each participant. The overall 
pattern indicates that most multiple-activity 
sequences involve job-search assistance 
coupled with either classroom training in oc- 
cupational skills or OJT. Our analysis con- 
trols for the differences in how SDAs record 
activities. The findings, therefore, should be 
interpreted as the impact on the percentage of 
participant services in various activities, in- 
cluding multiple activities. 

SDAs tend to report activities that are 
provided separately. In our site visits we 
found that many occupational skills pro- 
grams also included basic skills remediation 
or job-search assistance instruction that are 
not reported as separate activities. 

Impact of State Policies on Services 
Results for Adults. Table 23 presents the 
estimated impacts of State performance- 
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Table 22 



Means of Percentage of Services in Various Program Activities 
for SDAs Reporting Single and Multiple Activities 



Adults Youth 





Single 

Aprivitv 


Multiple 


Single 

Af fivitv 

imVp/U VilY 


Multiple 

Af tlVltlPQ 


Job-search assistance 
(including pre-employment/ 
work maturity training) 


20.1% 


26.0% 


21.1% 


24.6% 


Basic skills training 


6.7 


7.1 


9.3 


10.8 


Classroom training in 
occupational skills 


34.7 


30.8 


19.5 


15.8 


OJT 


33.4 


28.8 


18.1 


13.2 


Work experience 


2.6 


3.0 


12.4 


12.0 


Tryout employment 






15.9 


16.2 


Other activities 


2.6 


4.4 


3.7 


7.4 


n = 


113 


328 


113 


328 



Table 23 

Estimated Impact of State Policies on Percentage of JTPA Services 
in Various Program Activities— Adults 

(Standard Errors in Parentheses) 





lob-Search 
Assistance 


Basic Skills 
Training 


Classroom 
Training- 
Occupational 
Skills 


OJT 


Average 
Length of 
Participation 
(Weeks) 


Use of model (number of 


0.44 


-0.31 


-0.39 


0.09 


0.78*** 


standards for which model used) 


(0.75) 


(0.36) 


(0.67) 


(0.63) 


(0.23) 


.Adjustment procedures 


-4.34* 


1.99* 


2.73 


-1.21 


-1.98*** 


specified in policy 


(2.40) 


(1.13) 


(2.15) 


(2.00) 


(0!77) 


Emphasis on exceeding 


0.67 


-2.21*** 




-1.17 


0.15 


standards in 6% policy 


(1.49) 


(0.70) 


(L33) 


(1.24) 


(0.46) 


Use of 6% for hard 


-0.31 


-3.54*** 


4.66** 


2.60 


-0.52 


to serve 


(2.57) 


(1.21) 


(2.30) 


(2.14) 


(0.82) 


Policy for serving 


4 53 






r /10*** 


-1 "^4 
i.j't 


welfare recipients 


(2.87) 


(1.36) 


(2.57) 


(2.39) 


(0.88) 


Policy for serving 


5.06* 


0.35 


-0.21 


-4.16* 


1.77* 


dropouts 


(2.90) 


(1.37) 


(2.59) 


(2.41) 


(0.97) 


Incentive weight on cost-per- 


-0.13 


0.03 


-0.03 


0.20 


-0.13** 


positive-termination standard (%) 


(0.21) 


(0.10) 


(0.19) 


(0.18) 


(0.07) 


Incentive weight on 


-0.00 


0.03 


0.45** 


-0.43** 


0.05 


wage standard (%) 


(0.25) 


(0.12) 


(0.22) 


(0.20) 


(0.08) 



* Statisticu.iy significant at .10 level. 

* Statistically significant at .05 level. 

* Statistically significant at .01 level. 
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standards policies on the types of services 
received by adults in JTPA programs.* The 
DOL adjustment models do not adjust for ser- 
vices, and we find that the use of these 
models does not have uny significant impact 
on the types of services provided. SDAs that 
want an adjustment for providing a particular- 
ly costly program need to request adjust- 
ments beyond the model. The results indicate 
that SDAs in States that have such adjust- 
ment procedures enroll significantly more in- 
dividuals in basic skills training and enroll 
significantly fewer individuals in job-search 
assistance. 

Length of participation is included in the 
cost models, and use of the model does lead 
to significantly longer services. For example, 
SDAs in States that use the model for all 
seven standards are estimated to provide ser- 
vices lasting 5.8 weeks longer than SDAs in 
States that do not use the model at all. 
Policies specifying adjustment procedures, 
however, are unexpectedly associated with 
significantly shorter services despite the sig- 
nificant reduction in short-term job- search 
assistance. (One possibility is that SDAs in 
States without adjustment procedures may 
rely more on existing training programs, such 
as community colleges, that offer longer ser- 
vices at little expense to JTPA programs.) 

States that place greater emphasis on ex- 
ceeding standards tend to discourage the 
provision of basic skills services to adults. 
For example, SDAs in States that score low 
on this measure provide 2.8 percentage 
points more of their services in basic skills 
training than SDAs in States that score high 
on the emphasis-on-exceeding index. 
Policies that emphasize exceeding standards 
are associated with significantly more class- 



room training in occu national skills, 
however, rather than tlij^ less intensive job- 
search assistance. Thos, ti;ese policies appear 
to be increasing the employment focus of the 
program rather than reducing the intensity of 
services. 

SDAs in States that require that 6% 
money be used for the hard to serve also 
provide fewer of their services in basic skills 
training. This policy, however, increases the 
amount of classroom training in occupational 
skills. It is possible that the 6% funds allow 
SDAs to set up special purpose occupational 
programs for the hard to serve rather than to 
provide those participants with basic skills 
remediation only. 

In the analysis we included State policies 
for serving welfare recipients and dropouts 
because these two policies were found to be 
effective and because the JTPA legislation re- 
quires equitable service to these groups. 
Both policies significantly reduce the amount 
of services in OJT, the program activity as- 
sociated with the easiest-to-serve clients. 
Only the policy for welfare recipients, 
however, significantly increased the amount 
of basic skills training. 

Both policies for welfare recipients and 
dropouts tend to increase the amount of job- 
search assistance provided. Policies for serv- 
ing welfare recipients may increase job- 
search activities because of the common prac- 
tice of providing initial job- search assistance 
to welfare recipients as part of work-fare 
programs. However, these effects may also 
reflect the fact that SDAs often "make room" 
for more intensive services for some groups 
by providing very inexpensive services to 
others. The net result is that State policies for 
welfare recipients tend to shorten the length 



*The fraction of the variance in services explained by the models is significant but lower than for the client 
models. For adults, the R s of the initial models are .12 for job-search assistance, .13 for basic skills training, .23 for 
classroom training in occupational skills, and .28 for OJT. For youth, the R^s are .14 for job-search assistance. .15 
for basic skills training, .14 for classroom training in occupational skills, .25 for OJT, .09 for work experience, and 
.08 for tryout employment. 
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of participation, but State policies for 
dropouts significantly increase the average 
length of participation. 

The weights that States place on the dif- 
ferent performance measures also have some 
effect on the types of services provided 
SDAs in States that weight the cost-per- 
entered-employment standard more heavily 
provide significantly shorter training, sug- 
gesting that some SDAs react to the cost 
standard by reducing the intensity of ser- 
vices. On the other hand, placing greater 
weight on the wage-rate standard, a measure 
of job quality, significandy reduces the 
amount of OJT and significandy increases 
the amount of classroom training in occupa- 
tional skills. 

Results for Youth, Table 24 presents the 
estimated impact of State policies on tiie 
types of services received by youtii in JTPA 
programs. Use of the DOL adjustment 
models significandy reduces the amount of 
job-search assistance provided for youtii and 
increases tiie amount of basic skills training 
provided, altiiough not significantly. 
Procedures to adjust beyond tiie model do 
not significantly affect tiie types of services 
for youtii, although, as with adults, they have 
an unexpectedly negative impact on the 
length of program participation. 

SDAs in States tiiat place greater em- 
phasis on exceeding standards in tiieir incen- 
tive policies provide significandy less basic 
skills training foi youth. However, tiiese 
SDAs also provide significandy less job- 
search assistance (including pre-employ- 
ment/work maturity training) and 
significandy more classroom training in oc- 
cupational skills. This pattern suggests that 
policies emphasizing exceeding the standards 
intensify tiie employment focus of the youth 
program. This is consistent with tiie finding 
reported earlier tiiat such policies lead SDAs 
to focus then- youth programs on older youtii. 
Policies emphasizing exceeding standards 
also lead to significandy shorter program ser- 



vices for youth (perhaps because in-school 
programs tend to be a semester in length 
while out-of-school programs are often 
shorter). 

SDAs in States tiiat have policies for serv- 
ing welfare recipients offer significandy 
more basic skills training; SDAs in States 
witii policies for dropouts offer significandy 
less work experience and significandy more 
job-search assistance. Again, tiie latter effect 
may be because SDAs are "making room" 
for serving tiie more expensive dropouts. As 
with adults. State policies emphasizing ser- 
vice to dropouts lead to significandy longer 
services. 

SDAs in States tiiat place greater weight 
on tiie cost-per-positive-termination standard 
provide significandy more job-search assis- 
tance to youtii, which is tiie least expen- sive 
service, as discussed below. The weight 
placed on tiie youth entered-employment-rate 
standard, however, has no significant impact 
on program services. 

Summary of Impact of State Policies. 
State performance-standards policies do have 
significant impacts on program services. In 
general, policies tiiat adjust standards (tiiat is, 
use of models or additional adjustment proce- 
dures) tend to reduce tiie amount of job- 
search assistance and slighdy increase basic 
skills remediation, altiiough tiie impact on 
tiie length of training is unclear. State incen- 
tive policies tiiat emphasize exceeding stand- 
ards significandy reduce the amount of basic 
skills training and increase tiie amount of 
classroom training in occupational skills for 
both adult and youtii. The results suggest tiiat 
tiiese policies increase the employment focus 
of JTPA programs. 

Placing greater weight on tiie cost Stand- 
ards tends to reduce the intensity of JTPA ser- 
vices. Policies tiiat place more weight on tiie 
adult cost standard are associated witii 
shorter programs, and policies tiiat place 
more weight on the youtii cost standards tend 
to increase the amount of job-search assis- 
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Table 24 



Estimated Impact of State Policies on Percentage of JTPA Services 
in Various Program Activities— Youth 

(Standard Errors in Parentheses) 



Job-Search 
Assistance 
(Including 
Pre-employment/ 
Work Maturity) 


Basic 
Skills 


Classroom 
Training- 
Occupational 
Skills 


OJT 


Work 
Experience 


Try out 
Employment 


Average 
Length of 
Participation 


TI«f* of m(v1f*l ^niimher of 
Standards for which model used) 


-1.15** 
(0'.77) 


0.65 
(0.46) 


-0.10 
(0.56) 


-0.25 
(0.46) 


0.78 
(0.65) 


0.51 
(0.82) 


0.06 
(0.23) 


AHiiistmPTit nroTpHiinps 

Specified in policy 


-1 38 
(2.68) 


-0.48 
(1.60) 


1.49 
(1.97) 


-0.15 
(1.59) 


0.75 
(2.26) 


-2.15 
(2.87) 


-1.86** 
(0.82) 


Emphasis on exceeding 
standards m 6% pohcy 


-3.48** 
(1.45) 


-2.83*** 
(0.87) 


3.45*** 
(1.07) 


0.86 
(0.86) 


1.47 
(1.22) 


1.57 
(1.56) 


-1.04** 
(0.44) 


Use of 6% for hard 
to serve 


-0.24 
(2.80) 


-3.23* 
(1.68) 


0.64 
(2.06) 


-1.62 
(1.66) 


2.58 
(2.36) 


2.02 
(3.01) 


0.60 
(0.84) 


Policy for serving 
welfare recipients 


1.05 
(2.73) 


5.10*** 
(1.63) 


-1.05 
(2.01) 


-1.55 
(1.62) 


0.85 
(2.34) 


-3.17 
(2.98) 


-1.28 
(0.82) 


Policy for serving 
dropouts 


6.13* 
(3.13) 


-0.50 
(1.87) 


1.45 
(2.31) 


-2.92 
(1.86) 


-6.31** 
(2.64) 


2.61 
(3.16) 


3 "72*** 

(i".oi) 


Incentive weight on cost-per- 
positive-termination standard (%) 


0.56** 
(0.24) 


0.00 
(0.14) 


-0.27 
(0.18) 


0.04 
(0.14) 


-0.01 
(0.20) 


0.04 
(0.26) 


0.17 
(0.07) 


Incentive weight on entered- 
employment-rate standard (%) 


0.00 
(0.19) 


-0.07 
(0.11) 


-0.15 
(0.14) 


0.10 
(0.11) 


0.15 
(0.16) 


-0.03 
(0.20) 


-0.06 
(0.06) 



* Statistically significant at .10 level. 

** Statistically significant at .05 /el. 

*** Statistically significant at .01 level. 

ERIC J ' ^ 



Quantitative Analysis 



tance provided youth. Hov/ever, placing 
greater weight on wage-rate standard, a 
measure of job quality, significantly reduces 
OJT and increases classroom training in oc- 
cupational skills. 

State policies for serving welfare 
recipients (including additional State stand- 
aids) increase the amount of basic skills train- 
ing for both adults and youth. State policies 
for serving dropouts result in longer program 
services for both adults and youth, although 
these policies also increase the amount of job- 
search assistance provided. 

Impact of PIC and LEO Influence 

and Concerns on Services 

Results for Adults. Table 25 presents the 
estimated impacts of PIC and LEO influence 
and concerns on the types of program ac- 
tivities provided to adults in JTPA programs. 
Although the effects are not large, the pattern 
is consistent with that observed for enroll- 
ment of hard-to-serve clients. Specifically, 
PICs that are more influential in program 
design are associated with a significant in- 
crease in basic skills training. In contrast, 
PICs that are more influential in contracting 
process are associated with a significant in- 
crease in OJT services for adults. In general, 
PICs have a greater influence than LEOs. 

The effects on services of the PIC and 
LEO concern about performance are also 
similar to those reported for enrollment of 
hard-to-serve groups. Concern abour perfor- 
mance relative to the SDA's own standards 
has a relatively small and not significant im- 
pact on the types of program activity and, in 
fact, is associated with a significant increase 
in the length of program participation. Con- 
cern about performance relative to other 
criteria, such as performance of other SDAs 
in the State, however, is associated with sig- 
nificantly less basic skills training for adults. 

Results for Youth. Table 26 presents the 
estimated impacts of PIC and LEO concerns 
and influence on services for youth. As with 



the results for client groups, the pattern is 
less clear for youth than for adults. PICs that 
are influential in contracting are associated 
with a significant increase in OJT services, as 
was observed for adults. However, LEOs that 
are influential in program design are associat- 
ed with a significant increase in job-search 
assistance and, unexpectedly, a decrease in 
work experience. 

PICs and LEOs that are very concerned 
about performance relative to the SDA's own 
standards are associated with significantly 
less basic skills training for youth and sig- 
nificantly more classroom training in occupa- 
tional skills. This may again be a reflection 
of the belief that it is easier to meet standards 
in employment-oriented programs for youth. 

Summary of Impact of PIC and LEO. 
The impact of PIC influence on service 
provided is similar to that pattern found for 
the PIC influence on clients: PICs that are in- 
fluential in program design are associated 
with greater provision of basic skills training, 
services appropriate for some hard-to-serve 
clients; PICs that arc influential in contract- 
ing arc associated with greater provision of 
OJT to both adults and youth, services ap- 
propriate for easier-to-serve clients. Further- 
more, PIC and LEO concern about perfor- 
mance relative to other criteria, such as per- 
formance of other SDAs in the State, is as- 
sociated with less basic skills training for 
adults, although PIC and LEO concern about 
performance relative to the SDA's own stand- 
ards increases the employment focus of the 
youth programs. 

Impact of SDA Client 
Goals on Services 

As we discussed in Chapter V, planning 
goals for the types of clients to be served are 
hypothesized to affect the types of program 
services offered. Table 27 presents the es- 
timated impacts of SDA client goals on 
program services for both adults and youth. 
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Table 25 

Estimated Impact of PIC and LEO Concerns and Influence on Percentage of 
Services in Various Program Activities— Adults 

(Standard Errors in Parentheses) 





jODOeaTCn 

Assistance 




L^iassrooni 
Training- 
Occupational 

oiClUS 


UJ 1 


Average 
Length of 
Participation 


PIC influence on 


-U.oo 


1.07 


-l.lo 


^ AO 

-2.48 


0.02 


program design 


(1.88) 


(0.89) 


(1.68) 


(1.57) 


(0.59) 


PIC influence in 


0.4Z 


U.UZ 


-0.06 


2.60* 


-0.31 


contracting 


(1.59) 


(0.75) 


(1.42) 


(1.33) 


(0.50) 


LEO influence on 






f\ AO 

-0.48 


0.42 


-0.03 


program design 


(1.79) 


(0.84) 


(1.60) 


(1.49) 


(0.59) 


LEO influence in 


r\ on 

-u.yy 


U.o5 




-1.13 


0.09 


contracting 


(1.50) 


(0.71) 


(1.34) 


(1.25) 


(0.48) 


PIC and LEO concern 












about performance 












— Relative to standards 


0.74 


-0.06 


0.53 


-1.70 


1.06* 


set by State 


(1.75) 


(0.83) 


(1.57) 


(1.46) 


(0.55) 


— Relative to other 


-2.27 


-1.71* 


0.15 


1.82 


0.50 


criteria 


(2.08) 


(0.98) 


(1.86) 


(1.73) 


(0.65) 



* Statistically significant at .10 level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 



Table 26 

Estimated Impact of PIC and LEO Concerns and Influence on Percentage of 
Services in Various Program Activities — Youth 

(Standard Errors in Parentheses) 





Job-Search 
Assistance 
^Including 
Pre^mployment/ 
Wfirk Maturity) 


Basic 
Skills 


Classroom 
Training- 
Occupational 
Skills 


OJT 


Work 
Experience 


Try out 
Employment 


Average 
Length of 
Participation 


PIC influence on 
program design 


1.25 
(1.93) 


1.46 
(1.16) 


-1.54 
(1.43) 


-1.33 
(1.15) 


1.32 
(1.63) 


-2.31 
(2.08) 


0.33 
(0.60) 


PIC influence in 
contracting 


-1.31 
(1.63) 


-0.44 
(0.97) 


0.82 
(1.20) 


1.98** 
(0.97) 


-1.09 
(1.37) 


-1.76 
(1.74) 


-0.72 
(0.50) 


LEO influence on 
program design 


3.40* 
(1.87) 


-0.20 
(1.12) 


-0.03 
(1.38) 


0.84 
(1.11) 


-2.73* 
(1.58) 


-1.14 
(2.01) 


0.38 
(0.60) 


LEO ipiluence in 
contracting 


-0.26 
(1.55) 


0.09 
(0.93) 


0.82 
(1.14) 


0.20 
(0.92) 


1.27 
(1.31) 


0.05 
(1.67) 


-0.39 
(0.48) 


PIC and LEO concern 
about performance 
















— Relative to 
standards set 
by State 


-1.24 
(1.78) 


-1.77* 
(1.07) 


2.29* 
(1.31) 


-1.25 
(1.06) 


0.99 
(1.50) 


1.10 
(1.91) 


0.75 
(0.56) 


— Relative to 
other criteria 


-3.13 
(2.13) 


0.42 
(1.27) 


-1.56 
(1.57) 


0.69 
(1.26) 


0.40 
(1.79) 


0.75 
(2.28j 


0.65 
(0.66) 



* Statistically significant at .10 level. 

* Statistically significant at .05 level. 

* Statistically significant at .01 level. 



Table 27 

Estimated Impact of SDA Client Policies on the Percentage of Services in Various Program Activities 

(Standard Errors in Parentheses) 



Job-Search Basic Skills Classroom OJT Average 
Assistance Training- Length of 

Occupational Participation 
Adults Skills 



SDA established 
procedures to 
recruit and enroll 



Welfare recipients 


3.00 


-0.08 


1.51 


-2.61 


-0.24 




(2.51) 


(1.22) 


(2.26) 


(2.15) 


(0.82) 


Dropouts 


-0.98 


2.16** 


-2.91 


1.22 


0.32 




(2.11) 


(1.02) 


(1.90) 


(1.81) 


(0.68) 


Clients with other 


9,94*** 


-0.14 


-7 98*** 


-1.92 


-1 96*** 


barriers to employment 


(2.27) 


(1.10) 


(2'.04) 


(1.94) 


(0;75) 



Youth 


Job-Search 
Assistance 
(Including 
Preemployment/ 
Work Maturity) 


Basic 
Skills 


Classroom 
Training- 
Occupational 
Skills 


OJT 


Work 
Experience 


Try out 
Employment 


Average 
Length of 
Participation 


Welfare recipients 


-0.75 


1.43 


4.50*** 


1.56 


-3.17* 


-1.76 


0.19 




(2.23) 


(1.33) 


(1.63) 


(1.34) 


(1.89) 


(2.41) 


(0.69) 


Dropouts 


2.17 


4.27*** 


-0.62 


-0.66 


0.17 


-1.12 


0.1& 




(2.50) 


(1.49) 


(1.84) 


(1.50) 


(2.12) 


(2.70) 


(0.81) 


Clients with other barriers 


5.43** 


-2.01 


-3.00* 


0.37 


0.88 


-1.95 


0.52 




(2.31) 


(1.38) 


(1.70) 


(1.38) 


(1.96) 


(2.50) 


(0.74) 



* Statistically significant at .10 level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 
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Results for Adults. For adults, SDAs 
with policies to recruit and enroll welfare 
recipients do not provide a significantly dif- 
ferent mix of services, but SDAs that target 
dropouts provide significantly more basic 
skills training, a service very appropriate for 
that group. 

SDAs that target individuals with other 
barriers to employment provide significantly 
more job-search assistance and less class- 
room training in occupational skills* This 
result is consistent with our case studies find- 
ing that SDAs often conduct job-placement 
services for offenders leaving prison and also 
help develop appropriate jobs for hand- 
icapped individuals. Primarily because of the 
increased reliance on job-search assistance, 
the average length of participation is ap- 
proximately 2 weeks shorter m SDAs that 
target individuals with other barriers to 
employment. 

Results for Youth. SDAs that specifical- 
ly target youth welfare recipients offer sig- 
nificantly more classroom training in 
occupational skills, while those that target 
youth dropouts offer significantly more basic 
skills training. As is the case for adults, tar- 
geting those with other barritts to employ- 
ment is associated with significandy more 
job-search assistance and significantly less 
classroom training in occupational skills. 

Summary of Impact of Client Goals. 
These results indicate that there are sig- 
nificant relationships between the SDA goals 
for clients and the types of services offered. 
Generally, the results suggest that SDAs 
choose program services appropriate to the 
needs of the targeted clients; for example, 
SDAs that target dropouts provide more 
basic skills training. This important relation- 



ship between client goals and services 
provided is explored more in depth in the 
qualitative analysis. 

Program Costs 

The third outcome investigated in this 
evaluation is program costs, including expen- 
ditxue rates* We measure program costs as 
the average amount spent per terminee, as 
calculated from the JASR data.* In PY 86, 
the average cost per terminee is $2, 123 for 
adults and $1,924 for youth. We calculated 
expenditure rates from the SDA survey data; 
the average expenditure rate is 8l%* 

Impact of State Policies on Costs 
The estimated impacts of State perfor- 
mance-standards policies on adult and youth 
costs per terminee and on expenditure rates 
are presented in Table 28.** The adjustment 
models are intended to allow SDAs to spend 
more for hard-to-serve clients. However, the 
use of the DOL models does not si^iificandy 
affect program costs for adults or youth* Use 
of the models does significandy increase ex- 
penditure rates perhaps because SDAs are 
less averse to taking risks in their funding 
decisions. For example, SDAs in a State that 
uses the model for all seven standards are es- 
timated to expend 6.4% more of their avail- 
able funds than SDAs in States that do not 
use the models for any standard. 

State policies that specify procedures for 
adjustments beyond the model significandy 
increase program costs for both adults and 
youth. This may be because SDAs are 
granted adjustments to their cost standards 
or because SDAs are more willing to risk 
providing more expensive services if an ad- 
justment mechanism is in place should they 
have trouble meeting their cost standards. 
Adjustment procedures, however, are unex- 



*It is beyond the scope of this study io investigate SDA perfonnance as an outcome. Nonetheless, we also ex- 
amined the effects of policies and practices on the cost per entered employment and cost per positive termination. 
The pattern of results is very similar to the lesults present for costs per terminee. 

**The fraction of the variation explained by these initial models is .20 for adull costs, . 1 1 for youth costs, and 
.13 for expenditure rates. 
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Table 28 

Estimated Impacts of State Policies on Cost per Terminee 
and Expenditure Rates 





Adult Cost 
Per Terminee 


Youth Cost 
Per Terminee 


Expenditure 
Rates 


Use of model (number of 
standards for which model used) 


-14.3 
(22.7) 


6.2 
(22.2) 


(0'.4) 


Adjustment procedures specified 
in policy 


212.5*** 
(72.1) 


124.3* 
(72.3) 


-2.8* 
(1.5) 


Emphasis on exceeding standards 
in 6% policy 


111.1** 
(44.6) 


66.3 
(42.2) 


-1.4* 
(0.9) 


Use of 6% for hard to serve 


41.1 
(78.0) 


73.9 
(81.0) 


1.0 
(1.6) 


Expenditure rates incorporated 
into 6% award 


— 





1.3 
(1.6) 


Incentive weight on adult cost-per 
entered-employment standard (%) 


-3.0 
(5.9) 





-0.1 
(0.2) 


Incentive weight on youth cost-per- 
positive-termination standard (%) 




-3.6 
(6.0) 


0.2 
(0.2) 


Incentive weight on adult wage 
standard (%) 


8.3 

(7.0) 






Incentive weight on youth entered- 
employment-rate standard (%) 




0.7 
(5.3) 





* Statistically significant at .10 level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 
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pectedly associated with lower expenditure 
rates, perhaps because of delays in funding 
projects requiring adjustments. 

Earlier results indicated that State incen- 
tive policies that emphasize exceeding perfor- 
mance standards reduce service to several 
hard-to-serve groups, reduce provision of 
basic skills training, and increase provision 
of classroom training in occupational skills. 
The results in Table 28 indicate that policies 
emphasizing exceeding standards do not 
reduce costs per terminee but in fact increase 
costs, significandy so for adults. This pattern 
suggests that SDAs with strong incentives to 
exceed their standards focus their programs 
on easier-to-serve groups but provide train- 
ing in more expensive employment-oriented 
services. These policies, however, do not sig- 
nificantly increase either cost per entered 
employment or cost per positive termination, 
the cost measures that are rewarded with in- 
centive funds.* 

SDAs in States with policies that em- 
phasize exceeding standards do expend sig- 
nificandy lower amounts of tlieir available 
funds, perhaps because they are more 
cautious in committing fiinds to programs 
that may not perform well. However, these 
effects are not large. For example, SDAs in a 
State with a high score on this index are es- 
timated to expend 2% less of their funds than 
SDAs in States that score low on this index. 
State policies that incorporate expenditure 
rates into the incentive payments have only a 
small impact on expen(Uture rates that is not 
statistically significant. 

Surprisingly, the weights placed on the 
cost standards in incentive policies do not sig- 
nificandy reduce costs per terminee (nor do 
they reduce cost per entered employment or 
cost per positive termination). Furthermore, 
these weights do not affect expenditure rates. 

In summary. State performance-standards 



policies have fewer impacts on program 
costs and expenditure than on clients or ser- 
vices. Use of the model significandy in- 
creases expenditure rates. Adjustment 
procedures significandy increase the amount 
spent per terminee for both adults and youth, 
although unexpectedly decrease expenditure 
rates. The emphasis on exceeding standards 
actually increases the amount spent per ter- 
minee, significandy so for adults, suggesting 
that such policies lead SDAs to spend more 
on employment-oriented programs for less 
hard-to-serve clients. As discussed below, 
one reason that performance-standards 
policies have fewer effects on program costs 
is that there are serious problems in the com- 
parability of measured costs among SDAs be- 
cause of differences in the extent to which 
JTPA funds are leveraged with other funds. 

Impacts of PIC and LEO 

Concerns on Costs 

The estimated impacts of P^C and LEO 
concerns and influence on program costs are 
presented in Table 29. Again, these factors 
have less influence on program costs than on 
other outcomes. The PIC and LEO influen- 
ces have no significant impact on program 
costs. We do find that PICs and LEOs that 
are concerned about performance relative to 
standards tend to be associated with higher 
costs, significandy so for youdi, and diose 
concerned about performance relative to 
odier criteria tend to be associated widi lower 
costs, although die latter effects are not statis- 
tically significant 

Directors were also asked about the ex- 
tent to which PICs and LEOs were con- 
cerned about expending their available funds, 
and die results indicated diat diis concern is 
translated into significandy higher expendi- 
ture rates for the SD A. 



♦Policies emphasizing exceeding standards are estimated to increase cost per entered employment by $70 and 
the costs per positive termination by $4. Neither effect is significant. 
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Table 29 

Estimated Impacts of PIC and LEO Concerns and Influence on 
Cost per Terminec and Expenditure Rates 


f- 


Adult Cost 


Youth Cost 


Expenditure 


V- 


PerTerminee 


PerTerminee 


Rates 


iFIC influence on program design 


-31.3 


-21.8 


0^9 


r 


(57.9) 




n 1^ 


PlC influence in contracting 


-16 








(48.8) 


(48.6) 


(1.0) 


IFf) influence on ore cram design 


-37 0 


16 1 






(58.0) 


(58.2) 


(1.1) 


LEO influence in contracting 


43.1 


-15 3 


•l/.O 




(46.6) 


(46.3) 


(0.9) 


TIC and LEO concern about 








performance 








— Relative to standards set 


44.8 


97.5* 


0.6 


by State 


(54.4) 


(53.4) 


(1.1) 


— Relative to other criteria 


-54.2 


-50.7 


-0.0 




(63.4) 


(63.1) 


(1.3) 


PIC and LEO concern about 






2.0** 


Expending funds 






(0.9) 


f * Statistically significant at . 10 level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 
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Impacts of Service-Provider 

Arrangements on Costs 

The next step in the analysis is to ex- 
amine the impact of service-provider arrange- 
ments on program costs. An important issue 
for this evaluation is whether proj^ram costs 
reported by the SDAs reflect the crue costs of 
the services provided to JTPA participants. 
The hypothesis to be tested is that many 
SDAs leverage JTPA Title II- A funds with 
other resources so that the costs reported on 
the JASR represent only part of program 
costs. 

Although such leveraging may be a very 
valuable practice, the implication for the per- 
formance-standards system is that differences 
in costs among SDAs may reilect differences 
in the availability of alternative lunds to 
leverage with JTPA funds rather than dif- 
ferences in mar agement quality. If costs 
across SDAs are not comparable, then award- 
ing incentives based on exceeding the cost 
standard may introduce serious inequities 
into the performance-standards system. 

Directors were asked the percentage of 
their participants that were dual enrolled in 
other JTPA programs and the percentage that 
participated in programs that were also 
funded by other agencies. As shown in Table 
30, the total percentage of participants receiv- 
ing alternative funding is associated with 
lower reported program costs, significantly 
so for youth. SDAs were also asked about 
their expenditures m othtr JTPA programs 
(for example. Titles II-B and HI). SDAs that 
administer large JTPA programs in other tit- 
les also have significantly lower reported 
Title II-A costs. 

SDAs* choice of service providers also af- 
fects program costs, and some of these dif- 
ferences also reflect the extent that providers 
are subsidized by other sources. Programs 
provided directly by the SDA are more cost- 
ly, significantly so for youth programs. In 
contrast, providers that are public agencies, 
including schools and the employment ser- 



vice, are associated with lower reported costs 
to JTPA programs, although only the effect 
of the employment service on adult costs is 
statistically significant. Use of CBOs and for- 
profit providers have inconsistent impacts on 
program costs. 

The extent to which SDAs use perfor- 
mance-based contracts has virtually no im- 
pact on the cost per terminee for either adults 
or youth. As we discuss in the qualitative 
analysis, we found that SDAs have used a 
variety of mechanisms to build performance 
expectations into both performance-based 
and cost-reimbursement contracts to reduce 
program costs. 

The results in this section provide quan- 
titative support for the hypothesis that some 
of the measured differences in program costs 
across SDAs reflect differences in the extent 
to which JTPA Title II-A funds are leveraged 
mih other resources. These results indicate 
that the measured program costs may mis- 
represent both the total resources used to 
train JTPA participants and the differences in 
cost-effectiveness among SDAs. This issue is 
discussed further in the qualitative analysis. 

Impact of Program Activities on Costs 
The next step is to examine the relation- 
ship between program costs and the other 
two outcomes of this study, clients and ser- 
vices. As discussed previously, we believe 
that the strongest direction of causation is 
that the program activities offered affect the 
types of clients enrolled and that both of 
these factors affect the actual cost incurred 
by the program. 

We first examine the impact of program 
activities on program costs, presented in 
Table 31. The effects presented represent the 
estimated impact of increasing the percent- 
age of services in each activity relative to 
providing classroom training in occupational 
skills, the reference group. Job-search assis- 
tance is significantly less costly for both 
adults and youth than classroom training in 
occupational skills. For example, a program 
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Table 30 

Estimated Impacts of Service-Provider Arrangements on Costs 





Adult Costs 
PerTerminee 


Youth Costs 
Per Terminee 


Percentage of participants receiving funding 
from other programs or other JTPA sources 


-2.6 
(1.6) 


-2.6* 
(1.6) 


Expenditures in other JTPA programs as 
percentage of Title U-A 78% expenditures 


-2.8* 
(1.7) 


-3 2** 
(L6) 


Proportion of program activities 
provided by 






SDA itself 


60.4 
(123.3) 


201.5* 
(119.6) 


Public schools 

(including community colleges) 


-158.2 
(177.3) 


-57.4 
(173.8) 


Employment service and 
other government agencies 


-363.2** 
(143.5) 


-40.6 
(142.7) 


Community-based organizations 


-103.9 
(117.3) 


117.8 
(117.2) 


For-profit organizations 


181.1 
(139 8) 


-29.3 
(138.6) 


Percentage of expenditures in 
performance-based contracts 


0.7 
(1.5) 


-0.0 
(1.5) 



* Statistically significant at .10 level. 

* Statistically significant at .05 level. 

* Statistically significant at .01 level. 



12,9 



Table 31 

Estimated Impacts of Program Activities on Costs per Terminee 



Adult Costs Youth Costs 

Per Terminee Per Terminee 



Percentage of participants services 
in activities (compared with classroom 
training with occiq>ational skills) 



Job-search assistance 


-13.3*** 
(2.3) 


-12.5*** 
(2.8) 


Basic skills 


-4.3 
(4.1) 


-9 2** 
(3.6) 


on- 


-5.8** 
(2.8) 


-3.3 
(4.0) 


Work experience 


-3.1 
(5.3) 


-4.2 
(2.8) 


Tryout employment 




-10 2*** 
(2.6) 


Percentage of expenditures in 
supportive sen/ices 


(7^5) 


19 4** 

(?:?) 


Average weeks paiticipated 


8.2 
(5.3) 


3.8 
(5.0) 



* Statistically significant at. 10 level. 

* Statistically significant at .05 level. 

* Statistically significant at .01 level. 
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that is composed only of job-search assis- 
tance is estimated to cost $1,330 less than a 
program composed only of classroom train- 
ing in occupational skills. Basic skills train- 
ing is somewhat less expensive than 
occupational skills training, significantly so 
for youth. Because basic skills training is 
often provided by public schools for youth, 
this result may again reflect the leveraging of 
JTPA funds with other resources. OJT is also 
less expensive to provide, particularly for 
adults. For youth, tryout employment is also 
significantly less expensive than classroom 
training in occupational skills. 

The percentage of SDA expenditures in 
supportive services significantly increases the 
amount spent per terminee. Thus, SD As that 
provide support costs to participants do not 
simply reduce the amount spent in training 
but instead incur higher total costs. 

Finally, the average length of program 
participation has no significant impact on 
program costs. As discussed in the qualitative 
analysis, we found that several SDAs in our 
site visit sample paid tuition costs for par- 
ticipants receiving long-term training in com- 
munity college programs, institutions that 
receive substantial State funding. Thus, the 
lack of a relationship between length of par- 
ticipation and costs is probably due to the 
leveraging of JTPA funds in long-term 
programs. 

In summary, the results indicate that 
program activities significantly affect the 
costs per terminee in JTPA programs. For 
adults and youth, the most expensive service 
is classroom training in occupational skills 
and the least expensive is job-search assis- 
tance. 

Impact of Clients on Costs 

Table 32 presents die estimated impacts 
of serving different clients on program costs. 
These effects are estimated while controlling 



for the other factors examined for costs, in- 
cluding State performance-standards policies, 
PIC and LEO influence, service-provider ar- 
rangements, and program activities offei^.* 
After accounting for all tiiese factors, the 
types of clients served have relatively little 
impact on program costs. For adults, we ob- 
serve only two significant effects: SDAs tiiat 
serve more individuals who have been out of 
the labor force at enrollment incur higher 
costs, and SDAs that serve more dropouts 
spend significandy less per terminee. The lat- 
ter effect is surprising. SDAs may leverage 
funds more for dropouts, for example, by 
using public schools. However, some SDAs 
may be providing less intensive services to 
dropouts, such as job-search assistance. Both 
strategies were observed in our case studies. 

For youth, there are more significant dif- 
ferences. SDAs serving more youth welfare 
recipients spend significantly more per ter- 
minee. Serving in-school youth, particularly 
tiiose under 16, appears to be significandy 
less expensive to tiie JTPA system, probably 
because of heavy reliance on schools as 
providers. We also find that SDAs serving 
more youth who are dropouts spend less per 
terminee, similar to the result for adults. 

Summary and Conclusions 

This chapter examined the impact of 
State and SDA policies and practices on 
JTPA services and costs. Several State 
policies were found to affect the types of ser- 
vices offered by SDAs: 

• Adjustment procedures . Use of the 
DOL models significantly increases 
the length of adult programs and 
reduces the amount of job-search as- 
sistance provided to youth. State 
policies that specify procedures for 
additional adjustments significantly 
increase provision of basic skills train- 



*The characteristics of the eligible populaUon were not included in this model because those variables 
highly con-elated with the characteristics of clients actually served. 



Table 32 

Estimated Impacts of Terminees with Various Characteristics 
on Costs per Terminee 



Adult Costs Youth Costs 

Per Terminee Per Terminee 



Welfare recipients 1.4 52* 

(3.1) (3.2) 

Dropouts -10.7** -8 0* 

(5.1) (4.6) 

In school -14 3*** 

(4.2) 

Minorities -0.1 2 5 

(2.2) dig) 

Females I.9 11 

(4^4) (5.6) 

Other barriers to - 1 .5 33 

employment (3.7) (3.'l) 

Ages 30 to 54 5.6 _ 

(6.0) 

Ages 55 or older -49 

(8.7) 

Ages 14 to 15 — .9 4* 

(4.'8) 

Ages 18 to 21 — .6 5* 

(3.6) 

Not in labor force 57* 

(3.4) 

Unemployed 15 weeks 1.7 

or more (2.2) 



* Statistically significant at .10 level. 
** Statistically significant at .05 level. 
*** Statistically significant at .01 level. 



o 132 94 

ERIC 



ing to youth and reduce job-search 
assistance but have an unexpected 
negative impact on the length of 
participation. 

• Emphasis on exceeding standards . 
State policies that emphasize exceed- 
ing standards lead SDAs to provide 
less basic skills training and more 
classroom training in occupational 
skills for both adults and youth. For 
youth, these policies also reduce pre- 
employment/work maturity training 
and reduce the average length of 
program participation. 

• Weights on different standards . State 
incentive policies that place greater 
weight on cost standards reduce the 
average length of services for adults 
and increase the provision of pre- 
employment/work maturity training 
for youth. Emphasis on the wage-rate 
standard, a measure of job quality, sig- 
nificantly reduces the amount of OJT 
provided and increases the amount of 
classroom training in occupational 
skills. 

• State policies for hard-to-serve 
groups . State policies for serving wel- 
fare recipients significantly increase 
the amount of basic skills training of- 
fered for both adults and youth and 
reduce the amount of OJT, significant- 
ly so for adults. State policies for serv- 
ing dropouts significantly increase the 
average length of services for both 
adults and youth. State policies for 
serving dropouts also increase the 
provision of job-search assistance, 
reducing OJT for adults and work ex- 
perience for youth. 

The results of the previous chapter 
demonstrated that the types of program ser- 
vice have significant effects on the types of 
clients enrolled in JTPA programs. These 
results, which show that State policies affect 
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the types of services provided, indicate that 
the types of services offered are important 
mechanisms through which performance- 
standard policies affect clients. 

As was the case for clients served, the ef- 
fects of performance standards on services 
are significant and of policy importance but 
not so large as to preclude provision of ser- 
vices for which employment is not the im- 
mediate outcome. For example, on average, 
SDAs serve 7.0% of their auult clients in 
basic skills training and 31.8% in classroom 
training in occupational skills. 

An average SDA in a State with an incen- 
tive policy that places a low emphasis on ex- 
ceeding standards is estimated to serve 8.1% 
of adults in basic skills and 30.1% of adults 
in classroom training in occupational skills. 
A similar SDA in a State that places a strong 
emphasis on exceeding standards is estimated 
to serve 5.2% of adults in basic skills training 
and 34.6% of adults in classroom training in 
occupational skills. 

SDA characteristics and practices also 
affect the type of services offered: 

• PIC and LEO influence . A pattern 
similar to that observed for clients is 
evident for services. PICs that see 
their role as guiding the design of the 
program are associated with increased 
basic skills training for adults, while 
PICs that are more involved in con- 
tracting are associated with sig- 
nificantly more OJT services for both 
adults and youth. PICs and LEOs con- 
cerned with performance relative to 
the SDA*s own standards are as- 
sociated with signiflcantly longer 
program services for adults but with a 
stronger employment focus for youth. 
PICs and LEOs concerned about per- 
formance relative to other criteria, 
such as comparison with other SDAs 
in the State, are associated with sig- 
nificantly less basic skills training for 
adults. 
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• SPA policies for hard-to-serve 
groups . SDAs that have established 
specific procedures to recruit and en- 
roll dropouts provide significandy 
more basic skills training for both 
adults and youth. Policies for serving 
those with other barriers to employ- 
ment are associated with greater job- 
search assistance, probably because 
SDAs often conduct job development 
activities for offenders and the hand- 
icapped. 

This chapter also examined the impact of 
State performance-standards policies on 
program costs and found that performance- 
standards policies have fewer effects on costs 
than on clients and services: 

• Adjustment policies . Use of the DOL 
models has no significant effects on 
program costs, although State proce- 
dures to allow for adjustments sig- 
nificandy increase the amount spent 
per totninee for both adults and 
youth. Adjustment policies have an 
inconsistent impact on expenditure 
rates. 

• Emphasis on exceeding standards . 
State policies that emphasize exceed- 
ing performance standards increase 
costs per terminee, significandy for 
adults. Thus, tliese policies lead 
SDAs to enroll easier-to-serve clients 
but to provide them with more class- 
room training in occupational skills, 
the most expensive service. 

• Weights on different standards . The 
weight placed on the cost standard 
does not Significandy affect cost per 
terminee or SDA performance on the 
cost standards. 

One reason that there are fewer sig- 
nificant effects of performance-standards 
policies on program costs may be that some 
SDAs extensively leverage JTPA resources 



with other program resources. The ability to 
leverage resources may vary by State and 
thus confound our ability to distinguish the 
effects of State policies. 

Several SDA practices also affect 
program costs: 

• Service-provider arrangements . 
SDAs that dual enroll participants in 
other programs or receive direct fund- 
ing from other programs report lower 
costs per terminee. Furthermore, 
SDAs that administer other large 
JTPA programs report lower Tide 
II-A costs. The reliance on other 
public agencies as service providers 
also tend to lower program costs, al- 
though only the effect of using the 
employment service for adults is sig- 
nificant. The results provide quantita- 
tive support for the hypothesis that 
some of the differences across SDAs 
in reported costs reflect differences in 
leveraging JTPA fund with other 
resources. The use of performance- 
based contracts does not significandy 
affect program costs. 

• Program services . The types of 
program services provided significant- 
ly affect program costs. For both 
adults and youth, job-search assis- 
tance is the least-expensive service 
and classroom training in occupation- 
al skills is the most expensive. 

• Clients served Controlling for the 
types of services provided, the types 
of clients served have only a small im- 
pact on the amount spent per 
terminee. For both adults and youth, 
SDAs that serve more dropouts spend 
less per terminee, perhaps because of 
leveraging of funds with other resour- 
ces, such as public schools, or per- 
haps because they are given less 
intensive services. 
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Introduction 

In this chapter, we summarize the design 
of the qualitative component of the study and 
describe how this component was structured 
to address a specific set of research concerns. 
The first section of the chapter reviews the 
various goals of the qualitaive component 
The nz\l section describes the sampling 
strategy used to select the 8 States, 30 SDAs, 
and 87 service providers included in the case 
study data collection efforts. The third sec- 
tion of this chapter summarizes the types of 
respondents and outlines the topics covered 
in the onsite discussions in each case study 
site. The final section of this chapter 
describes the data analysis procedures that 
were used to distill the findings from the case 
study data. 

Goals of the Qualitative Component 

The qualitative component of the study 
has five major purposes: 

• To explore the influence of the 
Federal performance-standards sys- 
tem on States and SDAs; 



• To supplement the quantitative 
analysis in understanding how State 
performance-standards policies and 
practices influenced clients, services, 
and costs; 

• To understand the mechanisms by 
which both the Federal and the State 
performance-standards policies in- 
fluence clients, services, and costs; 

• To identify how the performance 
standards interact with a number of 
other factors in influencing State, 
SDA, and service-provider behavior; 
and 

• To address several additional study is- 
sues, including 

- How performance standards in- 
fluence the public image of JTPA 
programs, 

- Whether the technical assistance on 
performance-standards issues has 
been adequate, 

- Whether the local adjustment 
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models are adequate, 

- Whether the data collected for 
operating the JTPA system are of 
adequate quality, and whether the 
reporting burden is acceptable, and 

- What impact States and SDAs are 
anticipating from the implementa- 
tion of follow-up standards in 
PY 88. 

The indepth discussions with State, SDA, 
and service-provider staff during the onsite 
visits to the case study sites offered the op- 
portunity to trace how and why the perfor- 
mance standards affected program design 
and program implementation practices at all 
levels of the JTPA program. 

Assessing the Influence 
of the Federal Standards 

In assessing the influence of the Federal 
performance-standards system as a whole, 
we asked State and local staff, as well as 
State Job Training Coordinating Council 
(SJTCC) and PIC members, their opinions 
about the adequacy of the Federal standards 
in representing national program goals and 
about the influence of the Federal standards 
on State policy initiatives and SDA goal set- 
ting. At the service-provider level, there was 
little awareness of the Federal performance 
standards, so we traced how service pro- 
viders were reacting to the performance ex- 
pectations communicated to them by SDAs 
through goal statements and contract terms. 

Supplementing the Quantitative 
Assessment of State Policy Influence 
In tracing the different effects of varying 
Sta^e performance-standards policies, we 
wtrc interested not only in describing the key 
variations in State policy, but also in under- 
standing why States developed the policies 
that they did, how these policies were per- 
ceived by SDAs, and how the State policies 
shaped SDA and service-provider goals and 
practices. 
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Understanding the Mechanisms 
by Which the Standards Influenced 
Clients^ Services^ and Costs 
In describing the mechanisms by which 
the Federal and State performance-standard 
policies influenced JTPA clients, services, 
and costs, we traced the chain of effects 
through a number of discrete design and im- 
plementation phases, including 

• SDA goal statements about whom to 
serve. 

• SDA program-design go^^^ about the 
mix and intensity of se» * Cvj to offer, 

• SDA goals about what levels they 
wanted to achieve on the standards, 

• SDA and/or service-provider prac- 
tices in screening and enrolling 
clients, 

• SDA practices in selecting service 
providers and communicating perfor- 
mance goals and expectations to ser- 
vice providers, and 

• SDA and/or service-provider prac- 
tices relating to client placement and 
termination. 

Identifying How the Standards 
Interacted with Other Factors 
In describing how the performance stand- 
ards interacted with other factors in influenc- 
ing clients, services, and costs, we examined 
a number of factors, including 

• Characteristics of the local environ- 
ment, such as local SDA and service- 
provider goals and priorities, 
influence of historical service- 
delivery patterns dating back to the 
CETA program, local labor market 
conditions, and availability of dif- 
ferent types of service providers; and 

• Other State and Federal program 
regulations, such as equitable-service 
requirements for several subgroups, 
and the requirement to spend 40% of 
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all service expenditures on youth. 

Case Study Design: 
Sampling Strategy 

The design for the qualitative case studies 
was based on 3 nested samples, consisting of 
8 States, 30 SDAs (i SDAs in each of 7 
sample States and 2 SDAs in the last State), 
and 87 service providers (an average of 3 
providers in each SD A). The goals of the 
sampling design were 

• xo make the most effective use of 
limited resources, by clustering the 
study sites within a limited number of 
States, 

• To ensure that all three samples 
(States, SDAs, and service providers) 
would be representative of the Na- 
tion, and 

• To ensure that the SDA findings can 
be generalized from the SDA sample 
to the Nation. 

To makp each of the samples repre- 
sentative of the way in which the national 
JTPA funds have been expended, the prob- 
ability of selection at each level was propor- 
tional to the amount of JTPA funds expended 
by the unit of observation. At the State and 
SDA levels, this was formally built into the 
sampling procedures: the probability that 
each State and each SDA was selected was 
proportional to its PY 86 funding allocation. 
At the service-provider level, we selected the 
service providers with the largest contracts 
within each of the relevant agency and ser- 
vice types. 

State Sample 

The State sample was stratified according 
to two dimensions of State policies that were 
hypothesized to be important to the SDA 
response to performance standards: 

• The extent to which each State had 
policies or practices that emphasized 
services to hard-to-serve groups or 
population segments. 



• The e>:tent to which State 6% incen- 
tive award policies emphasized perfor- 
mance at the level of the standards 
versus performance in excess af the 
standards. 

Within each cell. States were drawn ran- 
domly, with the probability of selection 
proportional to allocation level. However, to 
ensure geographic diversity, only one State in 
each region was included in the sample. The 
resulting State sample included California, 
Maryland, Massachusetts, Missouri, New 
Mexico, New Yoik, North Carolina, and 
Ohio. 

Exhibit 2 describes some of the relevant 
features of performance-standards policies in 
these eight States, including 

• Whether the State has added addition- 
al performance standards beyond the 
Federal standards, 

• How "meeting a standard'' is defined, 

• How "exceeding a standard" is 
defined, 

• The number of standards required to 
be met before the SDA is eligible to 
receive any incentive awards, 

• Whether particular standards are 
given more weight in the State for- 
mula for rewarding SDA performance, 

• The extent to which 6% incentive 
funds are awarded for marginally ex- 
ceeding the standards, 

• Wiether incentive awards are 
"capped" at some level of perfor- 
mance beyond which higher perfor- 
mance yields no additional award, and 

• Whether performance on 6%-funded 
projects is considered to be excluded 
from the performance standards. 

Exhibit 2 also shows the size of the fund- 
ing allocation for PY 86 in each of the eight 
sample States (and the percentage of all 
JTPA funding) and the number of SDAs in 
each of the sample States (and the percentage 
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Exhibit 2 

Characteristics of Case Study States in PY 86 
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^ Beginning in PY 87, service levels to AFDC and dropouts are multipliers m the formula for 6% incentive funds. 
^ Beginning in PY 88, service to welfare, 
^ Beginning in P Y 88. meet 7/7. 
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3) percentage of PY 86 award given to the basis of marginally exceedhig the slandard. 
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of all SDAs in the Nation). 
SDA Sample 

Within each of the sample States, four 
SDAs were selected (except in New Mexico, 
where two of the three SDAs in the State 
were selected). As part of the selection 
process, all SDAs within each State were 
rank ordered by a summary index of their per- 
fomiance level in PY 85.* New York Qty 
was excluded from the sampling frame in 
New York, as was The Navajo Nation in 
New Mexico, because these SDAs are too 
unique to be representative of the Nation as 
a whole. After the sampling frame was 
prepared, one SDA was randomly selected 
from each performance, quartile, with the 
probability of selection proportional to the 
level of the SDA's PY 86 expenditures. In 
two States, the initial samples were discarded 
and redrawn because the initial sample did 
not result in sufficient geographic distribu- 
tion within the State. 

The resulting sample of 30 case study 
SDAs is shown in Table 33, along with addi- 
tional information about each SDA, including 

• The PY 86 funding allocation, 

• Total number of adult and youth ter- 
minees in PY 86, 

• The number of counties in the SDA, 

• The type of administrative entity 
(government, incorporated PIC, or 
nonprofit organization), 

• The percentage of PY 86 expendi- 
tures using performance-based con- 
tracts, if known, and 

• Unemployment rate (PY 86). 

The case study sample of SDAs also ex- 
hibited some variation in the extent to which 
they were successful in meeting the PY 86 
standards. Tw enty-three of the SDAs in the 



sample had met all seven Federal perfor- 
mance standards in PY 86. Of the seven case 
study SDAs that had miss^ J at least one 
standard, only one SDA was under threat of 
sanctions (for failure to meet four standards). 
One additional SDA had failed to meet two 
standards, while the remaining five SDAs 
had missed only a single standard. 

Table 34 compares the case study SDAs 
and the full sample of SDAs. The perfor- 
mance of the case study SDAs is very similar 
to all SDAs on all the youth standards and on 
all the aduit standards except the average 
wage at placement, where the case study 
SDAs achieved higher than average perfor- 
mance. In part, this is a reflection of some- 
what high average wage levels in the local 
economy. The major difference in the types 
of clients served is that the case study SDAs 
served substantially more minorities, which 
is largely a reflection of a substantially 
higher percentage of minorities in the eligible 
population in the case study sites than 
average. Differences in the other characteris- 
tics of the local area are minor. The types of 
program services offered by the case study 
SDAs are very similar to the average 
program services of all SDAs. Overall, the 
case study SDAs are quite representative of 
all SDAs. Those differences that occur 
generally reflect differences in the local area 
characteristics rather than differences in SDA 
behavior. 

Service-Provider Sample 

A sample of three service providers in 
each SDA was drawn by the field researcher 
conducting the site visit at the time the site 
visits were being arranged. In selecting 
providers, the objectives were 

• To ensure a range of variation in the 
types of providers sampled in that 



The performance index was constructed by comparing each SDA's actual performance on each of the seven 
Federal standards with its expected performance generated by the DOL local adjustment model. The difference be- 
tween the actual and expected performance was measured in standard deviations. TTie index was the average of this 
difference score across all seven standards. ^ 
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Table 33 
Case Study SDA Matrix 
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KEY: G = Governmental unit, P = Incorporated PIC. N = Nonprofit organization, 0 = Other type of organization 



Table 34 

Comparison of Case Study SDAs with all SDAs In PY 86 



Qualitative 
Sample * Full Sample* 



FY 86 Performance 
Adult 

Entered-employment rate 72% 72% 

Average wage at placement $5.32 $5.08 

Cost per entered employment $3,370 $3,368 



63% 



Welfare-entered-employment rate 63 % 
Youth 

Enttred-employmentrate 55% 52% 

Positive-termination rate 79% %q% 

Cost per positive termination $2,82 1 $2,759 

PY86 Terminee Characteristics 
Adult 

Welfare recipients 33% 29% 

Dropouts 28 26 

Minorities 57 45 

Youth 

Welfare recipients 30 25 

Dropouts 30 26 

Minorities 63 52 

Ages 18 to 21 63 62 

Characteristics of Eligible Population 
Adult 

Welfare recipients 35 33 

Dropouts 40 40 

Minorities 42 34 

Youth 

Welfare recipients 39 35 

Dropouts 22 21 

Minorities 47 39 

Ages 18 to 21 53 52 

continued on next page . . . 



* Weighted by PY 86 expenditures. 



Table 34 (Concluded) 



Qualitative 
Sample * Full Sample* 



Percentage ofPY 86 Participants 
In Various Program Activities 

Adults 

Job-search assistance 24% 25% 

Basic skills 6 7 

Classroom training in occupational skills 36 32 

OJT 28 28 

Youth 

Job-search assistance (including 

pne-employment/work maturity) 21 23 

Basic skills 13 n 

Classroom training in occupational skills 21 17 

OJT 14 13 

Work experience 6 12 

Ttyout employment 19 17 

Local Area Characteristics 

Unemployment rate 7.0 7.9 

Population density (1,000 per square mile) 1.6 1.4 

Percentage of families with income below poverty 1 0. 6 1 0. 8 

Average earnings ($l,O00*s) 20.0 19. 1 
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SDA (that is, educational institutions, 
community-based organizations, and 
other nonprofit or for-profit entities), 

• To ensure a range of variation in the 
types of services provided in that 
SDA (that is, OJT, basic skills remedi- 
ation, classroom training in occupa- 
tional skills, work experience/tryout 
employment, job-search assistance, 
and programs oriented to in-school 
youth), 

• To ensure a range of variation in the 
types of contracts used in the SDA 
(that is, cost reimbursement and per- 
formance based), 

• To pick the contractor with the largest 
contract within each type of training, 
and 

• To select service providers that were 
within a 2-hour drive of the city in 
which the SDA headquarters was 
located. 

Because it was impossible to represent 
all types of services and providers with only 
three providers in each SDA, the service- 
provider sample was selected purposively 
rather than randomly. However, we were suc- 
cessful in generating a sample of 87 provi- 
ders that represents all of the different types 
of services and providers across the 8 States 
and 30 case study SDAs. (In several SDAs, 
fewer than three service providers were inter- 
viewed because the SDA contracted with 
fewer than three entities for the provision of 
services.) 

Rather than analyzing the service- 
provider sample separately, we integrated 
data from the service-provider sample into 
data from the SDA sample to inform the dis- 
cussion of implementation practices and out- 
comes within each case study SDA. Exhibit 3 
summarizes the service-provider sample and 
types of service providers across all eight 
States. The complete list of providers is 



presented in Appendix B. 

Among 87 providers, 23 (26%) were 
public educational institutions; 7 (8%) were 
private educational institutions (3 of which 
were nonprofit and 4 of which were for- 
profit); 12 (14%) were other public agencies; 
9 (10%) were SDAs offering one or more ser- 
vices directiy; 20 (23%) were community- 
based organizations; 12 (14%) were other 
nonprofit organizations; and 4 (5%) were 
other for-profit organizations. 

The most firequendy provided service 
among the service providers in the sample 
was classroom training m occupational skills, 
offered by 44 of the 87 service providers, fol- 
lowed in frequency by OJT, offered by 25 
service providers. Less frequendy offered ser- 
vices included basic skills remediation, of- 
fered by 20 service providers, and job-search 
assistance, offered by 16 service providers. 
Work experience or tryout employment were 
offered by 10 providers. Eleven of the 
providers in the case study sample offered 
YEC-oriented training. Three offered sup- 
ported employment to disabled target groups. 
Nine of the contractors (other than the SDAs 
themselves) had responsibility for centralized 
intake, assessment, and/or eligibility deter- 
mination under contract to the SDA. 

A total of 79 contracts were used in PY 
86 with the 77 service providers that were 
under contract to SDAs. (Nine service 
providers were themselves SDAs, and one ad- 
ditional service provider was another city 
agency with a memorandum of under- 
standing rather than a contract with the 
SDA.) Of the 79 contracts, 35 were cost-reim- 
bursement contracts, and 44 were perfor- 
mance-baGcd contracts. Of the performance 
-based contracts, only 13 held back more 
than 50% of the total contract amount for 
placement, while 31 paid out more than 50% 
of the total contract amount for payment 
points prior to placement. 
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Exhibit 3 



Summary of Characteristics of Service-Provider Sample by Types of Service Provided 

(N=87) 
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Case Study Design: Data 
Collection Methods 

The data collection methods used in the 
qualitative study component included inter- 
views with State, SDA, and service-provider 
respondents and reviews of written docu- 
ments, including SDA plans and service- 
provider contracts. Onsite visits to each 
sample site included 1 day of interviews at 
the State level and 2 to 3 days of interviews 
at the SDA level. 

Respondents 

At the State level, interviews were con- 
ducted with 

• SJTCC staff; and 

• Staff of the State agency administer- 
ing JTPA programs, including 

- Policymakers, 

- Performance standards technical 
staff, 

- SDA monitors, and 

- MIS and evaluation staff. 

At the SDA level, interviews were con- 
ducted with 

• PIC representatives, 

• SDA directors, 

• SDA planners and policymakers, 

• Staff who negotiate contracts with 
service providers, 

• Staff who monitor service-provider 
performance, and 

• MIS and evaluation staff. 

At the service-provider level, we talked 
with program administrators and direct ser- 
vice staff to understand how individuals in 
each of these roles perceived the performance 
expectations placed on them by the SDA and 
how that influenced their decisions in plan- 
ning and operating a service program. 

When interviewing a particular in- 
dividual, we discussed each topic about 
which he or she was informed. This meant 



that a number of topics were discussed with a 
variety of respondents at different levels of 
authority and organizational affiliation. For 
example, we heard about contract terms and 
how they affected service-provider practices 
from both the service provider's and SDA*s 
perspectives. 

Case Scudy Issue Topics 

Exhibits 4 to 7 summarize the topic areas 
that were covered in discussions with State 
staff, PIC representatives, SDA staff, and ser- 
vice-provider staff, respectiveiy. Complete 
copies of the discussion guides used during 
the site visits are included in Appendix E. 

Case Study Design: Data 
Analysis Procedures 

Once the field visits were completed, we 
prepared detailed narrative summaries of 
each State and each SDA for our own inter- 
nal use. Cross-site analyses were performed 
by observing similarities and differences in 
the patterns exhibited by the different SDAs 
and taking into account their responses to 
the standards and other external factors. In- 
dividual staff responsible for summarizing 
fundings for a particular topic area developed 
detailed abstracts of the fmdings across the 
30 case study sites in that topic area and 
described the resulting patterns. 

The typologies described in the next chap- 
ter grew out of this analysis process as pat- 
tans emerged and were found to be useful 
tools in describing the various ways SDAs 
reacted to the performance standards. These 
typologies are intended to simplify the diver- 
sity of SDA behavior in the real world in 
order to highlight key differences, rather than 
to accurately describe the actual behavior of 
any single SDA or group of SDAs. In par- 
ticular, the "types" of SDAs described in 
Chapter VIU represent extreme positions or 
"pure types" on their particular dimensions, 
with most real-world SDAs located on a con- 
tinuum between these extreme positions. 

The fmdings generated from the qualita- 
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tive component are intended to be gener- 
alizable to the Nation in a limited sense. The 
case study samples represent the full range of 
variations on a number of critical variables, 
including the size of SDA allocation, local 
economic conditions, State performance- 



standards policies, and prior SDA perfor- 
mance levels. Thus, we are confident that the 
case study findings do reflect a broad range 
of variations in SDA response to the study 
sample should not be extrapolated to the Na- 
tion as a whole, except as a general indica- 
tion of the distribution of responses. 
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Exhibit 4 

Outiine of Topics for Onsite Discussions: State Staff 
Qualitative Analysis 

A. Roles and Responsibilities of Different State-Level Actors 

B. Summary Views on Performance Standards 

C State Views on Performance-Standards Design (for example, the levels at which Federal 
standards are set, the adequacy of local adjustment models, additional State standards, 
and rewards and sanctions policy) 

D- State Views on Other Federal Policies Affecting the JTPA System (for example. Federal 
limits on administrative costs and supportive service costs) 

E. Other State Policies Affecting the JTPA System 

F. State Reactions to SDA Performance in FY 85 
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Exhibits 

Outline of Topics for Onsite Discussions: PIC Representatives 
Qualitative Analysis 

A. Role of the PIC 

B. SDA Goals on Clients Served 

C. SDA Goals on Service Design 

D. SDA Development of YEC System 

E. SDA Performance Objectives 

F. PIC Opinion About Performance 

G. Summary Views of Performance Standards 
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Exhibit 6 

Outline of Topics for Onsite Discussions: SDA Respondents 
Qualitative Analysis 

A. SDA Goals, Objectives, and Priorities 

1 . Roles of Different Local Actors 

2. Effects of Local Factors on SDA Performance Objectives and Practices 

3. Summary Views on Performance Standards 
4 SDA Goals on Clients Served 

5. SDA Goals on Service Design 

6. SDA Development of YEC System 

7. SDA Suggestions for Improving the YEC System 

B. Development of SDA Performance Objectives 

1 . SDA Performance Objectives 

2. SDA Strategies for Improving Performance from Year to Year 

C. SDA Policies and Practices for Program Implementation 

1 . Description of the Flow of Clients 

2. Practices Affecting Client Selection 

3. Practices Affecting Services Delivered 

4. Practices Affecting Types of Service Providers and Regulation of Provider 
Performance 

5. SDA Practices and Procedures Regarding Placement and Termination 

D. Understanding of Federal and State Performance-Standards Policies 

1 . Choice of Additional S tate Performance Standards 

2. State Alterations/Exceptions to the Local Adjustment Models 

3. State Design of the Incentive Rewards System 

4. State Sanctioning Actions 

5. SDA Actions Regarding Utilization of State Rewards for Past Performance 

6. SDA Actions Regarding Receipt of State 6% Funds Set Aside for Hard-to-Serve 
Clients 

E. Impact of Other Federal and/or State Policies on SDA Policies and Practices 
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Exhibit? 

Outline of Topics for Onsite Discussions: Service Providers 
Qualitative Analysis 

A. History of Service Provider as an Employment Services Contractor 

B. Contractual Relationships/Relative Roles of SDA and Service Provider 

C. Service-Provider Priorities, Incentives, and Policies 

D. Description of Service-Provider Practices 

1 . Describe Service-Provider Practices Regarding Client Outreach 

2. Describe Service-Provider Practices Regarding Client Assessment, Selection, 
Enrollment, Assignment to Services, and Provision of Supportive Services 

3. Describe Placement, Termination, and Follow-up Practices 

E. Effect on Service Provider of SDA Policies Regarding Client Priorities and Client 
Selection 

F. Effect on Service Providers of SDA Policies Regarding Service Design and Assignment 
to Service Issues 

G. Effect on Service Providers of SDA Policies Regarding Acceptable Service Costs 

H. Issues Relating to Youth Competency Areas (For Providers Serving Youtli) 

I. Effects of Local Factors on Service-Provider Practices and Performance 
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VIIL Overview of SDA Strategies and Goals: 
Evidence from the Qualitative Analysis 



Introduction 

The previous chapters summarizing the 
quantitative analysis have found that State 
performance-standards policies do significant- 
ly impact both the clients enrolled and the ser- 
vices offered in JTPA programs. Generally, 
policies that emphasize exceeding standards 
tend to reduce service to hard-to-serve 
groups and increase the provision of services 
designed to achieve immediate employment; 
policies that emphasize service to hard-to- 
serve groups and that adjust standards for 
serving such clients tend to increase service 
to these groups and increase the intensity of 
program services. The quantitative analysis 
also produced evidence that the effects of per- 
formance standards on clients are produced, 
in part, through their impacts on services of- 
fered and, in part, through their impacts on 
SDA management practices, such as client 
screening and referral procedures. 

We turn now to the results of the qualita- 
tive evaluation, based on extensive case 



studies in 30 SDAs. This component comple- 
ments the quantitative analysis by examining 
how Federal and State performance-stand- 
ards policies influence SDA choices, includ- 
ing formulation of program goals, design of 
program services, and practices to implement 
and manage the JTPA program. The qualita- 
tive analysis clearly demonstrates that the 
quantitative results represent national 
averages of highly varied responses to the 
performance-standards system. Before 
presenting the qualitative results on the im- 
pacts of performance standards on clients, ser- 
vices, costs, and management practices in the 
subsequent chapters, this chapter provides an 
overview of how SDAs differ in their respon- 
ses to performance standards and how this af- 
fects their performance goals. 

Overview of Chapter 
and Key Findings 

The first part of this chapter describes 
typologies of the SDAs' goals and objectives 
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that interact with Federal and State policies. 
We developed these typologies to help ex- 
plain the variations in how performance 
standards influenced clients, services, and 
costs within the case study sample. 

Client-oriented SDAs gave priority to 
tlieir client objectives and oriented their 
program design and management decisions 
around how this would affect their client 
goals. Employer-oriented SDAs gave priority 
to designing and implementing a service mix 
that would train individuals for jobs in high 
demand in the local labor market. Perfor- 
mance-oriented SDAs gave priority to meet- 
ing or exceeding the performance standards 
(or other State or local performance 
measures). These were SDAs without strong 
employer or client goals to counterbalance 
their goals to exceed their performance stand- 
ards. However, most SDAs could meet their 
performance goals without having to resort 
to a major revision of other program goals. 

Other typologies are described; one is the 
extent to which SDAs managed their risk or 
avoided risk with respect to the standards, 
and the other is the extent to which SDAs 
shared the risk of meeting the performance 
standards with their service providers. 

The next part of the chapter describes the 
variation among the case snidy SDAs in the 
extent to which the performance standards 
have influenced local program goals, 
program design, and management practices . 
About a third of the SDAs had a goal of 
meeting the standards and "staying out of 
trouble" (that is, avoiding sanctions), another 
quarter wanted to exceed the standards slight- 
ly, to have a performance safety r^argin, and 
to realize some incentive awards, and the 
remaining sites had a goal of performing at 
as high a level as they could on the stand- 
ards. The relative importance of the perfor- 
mance standards was conditioned in part by 
the importance that SDAs placed on receiv- 
ing incentive awards. 



The final section of the chapter describes 
a number of other factors that were important 
influences on SDA strategies in the case 
study sites. These include local factors such 
as the local unemployment rate and labor 
market opportunities, as well as the 
availability of service providers, and State 
and Federal factors such as the 40% youth ex- 
penditure requirement, the equitable service 
requirements for dropouts and welfare 
recipients, and the 15% limit on supportive 
services costs and lack of stipends. These en- 
vironmental factors influenced the SDAs' 
program-design and client selection priorities 
as much or more than did the performance 
standards. 

Typologies from the 
Qualitative Analysis 

The typologies presented below grew out 
of the analysis of the 30 case studies; they 
represent a synthesis of the interviews con- 
ducted with SDA and service-provider staff. 
No SDA perfectly meets the description of 
any one of these types, which arc exag- 
gerated "pure types" of tendencies found in 
more blurred form in the real worid. The 
categories described here were developed to 
provide a framework for describing the kinds 
of variations in goals, objectives, and prac- 
tices that actually existed among the case 
study SDAs and as a guide to the interpreta- 
tion of the variations. However, these 
typologies oversimplify and exaggerate the 
complex features observed in the case study 
sample to highlight the key variations in 
SDA response to the performance standards. 
For this reason, we have not attempted to 
count the number of case study SDAs in each 
category. 

"Client" Versus "Employer" Versus 
"Performance" Orientation 

The first way to categorize SDAs is to 
array them by the amount of emphasis that 
they place on three "pure" types of program 
objectives: (1) a commitment to serving 
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specific types of clients , (2) a commitment to 
responding to local employer needs and inter- 
ests, and (3) a commitment to achieving 
specific levels on the JTPA performance 
standards . In some SDAs, all of these objec- 
tives are present simultaneously; in others, 
only one or two of these objectives have 
received formal emphasis or attention. SDAs 
with differing amounts of emphasis on these 
objectives will differ in how they balance 
decisions about program design and service 
mix, recruitment and client selection prac- 
tices, and program management practices, 
such as provider selection and contract terms, 
enrollment decisions, and reporting practices. 

The experiences of the case study sites 
suggest that the more client oriented a given 
SDA is, the more the SDA will give priority 
to its client objectives and evaluate its 
program-design and management decisions 
according to how they will affect the achieve- 
ment of its primary client-targeting goals. 
The more employer oriented a given SDA is, 
the more the SDA will give priority to design- 
ing and implementing a service mix that will 
train individuals for the more highly skilled 
jobs for which employers need trained 
workers, and the more this objective will in- 
fluence decisions about whom to enroll in 
training programs and what performance 
measures to emphasize. Finally, the more per- 
formance oriented a given SDA is, the more 
the SDA will be willing to compromise client 
and service goals to achieve high perfor- 
mance levels. 

In the following chapters, we have some- 
times referred to this typology in explaining 
the variation in SDA reaction to the perfor- 
mance standards. The typology is useful be- 
cause it illustrates the tension between client, 
service, and performance objectives present 
in many SDAs and shows how the perfor- 
mance standards interact with other SDA ob- 
jectives in influencing decisions about 
clients, services, and costs. 
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The analysis indicates that performance 
standards have their greatest unintended ef- 
fects on program-design and implementation 
practices in SDAs that are lacking in strong 
client or employer orientations to counter- 
balance their performance concerns. In con- 
trast, the performance standards have the 
least unintended effects in SDAs with strong 
client or employer objectives that are the 
primary iniluence on program-design 
decisions and that reduce the influence of per- 
formance concerns. 

^^Risk Management" Versus 

'^Risk Avoidance" Strategies 

A related typology that is useful in high- 
lighting variations among SDAs in how they 
have responded to the performance standards 
is the difference between SDAs that have 
developed strategies of "risk avoidance" ver- 
sus "risk management'* to achieve their per- 
formance objectives. 

One type of SDA is "indifferent to the 
performance standards " in developing their 
program-design and client-mix strategies. 
These SDAs do not pay much attention to the 
performance standards, either because they 
are not highly motivated to meet or exceed 
the standards or because the Lvels at which 
the standards have been set (or maximum in- 
centives are earned) are very easy for them to 
meet. 

At the opposite end of the spectrum are 
SDAs that arc "risk avoiders ." In this type of 
SDA, concem about meeting or exceeding 
standards takes priority over other objectives 
(as in the "performance-oriented" SDAs 
described above). The SDA's design 
decisions and client recruitment practices are 
chosen to ensure that its performance objec- 
tives will be met. For example, the SDA will 
avoid serving individuals with serious 
employment barriers because there is con- 
siderable uncertainty about whether those in- 
dividuals will be successful in completing the 
JTPA program and obtaining employment. In 
"risk-avoiding" SDAs, the SDA*s client and 
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employer objectives are amended if they 
create undue risks for the achievement of the 
SDA's performance objectives. 

A third type of SDA, the * *risk manager ", 
falls between the two extremes. "Risk 
manager" SDAs are concerned about achiev- 
ing performance objectives but have also 
developed management practices that will en- 
sure that they will meet their performance 
goals while still furthering their local client 
and service objectives. The strategies usually 
involve acknowledging that various service 
programs and client groups will have dif- 
ferent outcomes and balancing the overall 
program design so that performance goals 
can be reached. 

Because both successful risk avoidance 
and successful risk management lead to high 
performance on the standards, both strategies 
tend to be rewarded by the performance- 
standards system. However, risk avoidance 
strategies generally amount to taking the 
easy way out, emphasizing the standards as 
the sole indicators of success while ignoring 
the fact that there may be other important 
local or national objectives (for example: 
reaching individuals who have serious 
employment barriers or achieving stable 
employment outcomes). SDAs using risk 
management strategies recognize that there 
are organizational goals worth pursuing be- 
sides high performance. Risk management 
SDAs try to find ways to accomplish their 
desired performance on the standards while 
still furthering these other goals. 

"Risk Sharing" Versus "Risk 
Passing'*: SDA Versus Service- 
Provider Performance Responsibilities 

A final typology useful in understanding 
the variations among the case study SDAs 
describes the extent to which the SDA as- 
sumes responsibility for meeting its perfor- 
mance goals, shares that responsibility with 
service providers, or passes on the primary 
responsibility for performance to the service- 
provider level. SDAs that keep the risk of 



performance at the SDA level of administra- 
tion may contract for training services with 
other organizations (or to operate programs 
directly) but do not require that service 
providers perform at a certain level to receive 
full compensation. This type of SDA 
depends heavily on careful planning of 
program mix for the entire SDA and careful 
monitoring of service-provider and SDA per- 
formance throughout the year. Usually the 
SDA tries to keep control of some mech- 
anisms that it can use near the end of the year 
to fine tune the SDA*s overall performance 
(for example, providing additional funding to 
high-performing projects and reducing the 
duration of training contracts that are ap- 
proved late in the year). 

A second type of SDA shares perfor- 
mance risks with its service providers. This 
type of SDA usually sets forth very specific 
performance goals for its service providers 
(through the terms of either cost-reimburse- 
ment or performance-based contracts) and 
takes clear steps to reduce funding or not 
retain contractors that do not live up to their 
performance expectations. However, "risk- 
sharing" SDAs also often share with their ser- 
vice providers the responsibilities for 
screening and enrolling clients and some- 
times for placing clients at the conclusion of 
training. The SDA and service providers try 
to work together to achieve the SDA*s perfor- 
mance objectives. 

The third type of SDA passes the risk 
along to the service-provider level through 
clear performance-based contracts, with most 
of the payment reserved for achievement of a 
job placement or job retention outcome. In 
this type of SDA, the performance standards 
have the potential for having the greatest 
unintended effects on the types of clients 
served by JTPA programs. If service 
providers have discretion over who is en- 
rolled in the program, they are allowed to 
determine the tradeoff between client and per- 
formance objectives while they are given 
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findJicial incentives to weight performance 
objectives over client objectives. 

Importance of the Standards in 
Shaping Local Performance 
Objectives and Strategies 

We did not categorize the SDAs visited 
according to the typologies developed above, 
because the typologies represent pure or ideal 
types. Rather, we developed the typologies to 
understand why many SDAs reported that the 
performance standards did not have a great 
influence on the way they did business. Other 
SDA objectives, such as meeting client or 
employer needs, often had higher visibility 
than the performance-standards themselves. 

There was considerable variation among 
the 30 case study SDAs in both the extent to 
which and the ways in which the perfor- 
mance standards influenced local program 
goals, program design, and management prac- 
tices. One important distinction among SDAs 
concerns their specific performance goals. 
Approximately 1 1 of the 30 case study SDAs 
(37%) had a goal of simply meeting the stand- 
ards and "staying out of trouble" (that is, 
avoiding sanctions). Another seven SDAs 
(23%) indicated a goal of exceeding the 
standards slightly, to have a performance 
safety margin and to realize some incentive 
awards. Finally, 12 of the 30 case study sites 
(40%) indicated a goal of performing at as 
high a level as they could on the performance 
standards, because it was important to them 
to maximize their potential 6% incentive 
award. Pride and good public relations were 
also mentioned by all types of SDAs as im- 
portant incentives to perform at or above the 
performance levels set by the standards.* 

Many SDAs with strong performance 
goals, however, also stated additional client 



and service goals that were important to them 
and that moderated the concern for high per- 
formance on the standards in some cases. Ap- 
pendix D sununarizes the performance goals 
of all 30 case study sites (without identifying 
individual SDAs by name). 

Importance of Incentive Funds 

The development of performance goals 
and the relative importance of the perfor- 
mance standards in shaping other local goals 
and priorities was affected in part by the im- 
portance the SDA placed on receiving the 
maximum amount of incentive funds. SDAs 
varied in the extent to which they attached 
importance to receiving incentive funds. 

In about half of the case study SDAs, the 
amount of incentive funds received was a 
matter of great importance. The i^asons for 
emphasizing incentive awards varied but prin- 
cipally consisted of the following: 

• Many SDAs, particularly those in 
areas of declining unemployment, 
needed the money to compensate for 
shrinking 78% allocations (10 SDAs, 
30%). These funds were needed to 
maintain the level at which programs 
were operating. Several SDAs valued 
the flexibility to use 30% of the incen- 
tive award for administrative expendi- 
tures (mentioned specifically by 8 
SDAs, 27%). 

• SDAs often attached a great deal of 
importance to performing well com- 
pared with other SDAs in the State, 
being number one in the State (as 
measured by the size of the award), or 
receiving the maximum incentive 
funds for which they were eligible, as 
a matter of pride. Three SDAs (10%) 
additionally cited the public relations 



♦Throughout the remainder of the qualitative analysis, we often indicate llie number and percentage of SDAs in 
a certain category or giving a certain response. Although the SDA sample was selected to be as rq)resentative as 
possible, due to the small sample size these percentages should not be used to extrapolate to ihe Nation as a whole. 
They are intended as a convenience to the reader only. 
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value (in the local community) of 
receiving "an award" as more impor- 
tant historically than the money itself. 
(However, some said that the funds 
themselves are now becoming increas- 
ingly important 78% funding 
shrinks.) 

• In two SDAs (7%), respondents 
stated that they saw tlie incentive 
funds as an opportunity to undertake 
projects that they could not fund with 
78% money, either because they were 
high risk (serving individuals with 
multiple barriers to employment) or 
because they only indirecUy con- 
tributed to measured outcomes (for 
example, funding special efforts to in- 
crease outreach or marketing to 
employers). The SDAs that men- 
tioned this factor were among those 
that were permitted by their States to 
exempt 6% funded projects from the 
performance standards — which was 
not typical at the time of the case 
study visits. 
In the remaining half of the SDAs, the in- 
centive award was less important. In fact, 
two SDA respondents stated definitely that 
the money was not of interest. For some 
SDAs, the amount of money at stake was so 
small that it was hardly noticed. For others, 
client recruiting was difficult, and thus, they 
were already having trouble expending their 
78% allocation. SDAs that operate other 
programs in addition to JTPA programs 
generally attached far less importance to the 
6% funding programs because they had other 
ways of compensating for declining alloca- 
tions and administrative funding constraints. 
Additional reasons for attaching relatively lit- 
tle importance to incentive awards were the 
following: 



• The unevenness or unpredictability of 
the funding made it difficult to plan 
sensible ways to use it (two SDAs, 
7%). 

• For some SDAs, the incentive fund- 
ing arrived with even more con- 
straints than the 78% allocation: they 
needed to spend it quickly and thus 
had to undertake a midyear planning 
effort around a relatively small sum 
and had to be accountable (via the 
performance standards) for outcomes 
achieved with the funds. 

Uses of Incentive Funds 

Incentive funds were spent in three dif- 
ferent ways: 

• At least eight SDAs (27%) used the 
full 30% allowable for administrative 
activities and considered this an im- 
portant feature of the 6% funds. 
Sometimes the additional administra- 
tive "cushion" allowed SDAs to use 
the funds in unusual ways. Three 
SDAs gave staff raises or bonuses 
based on the amount of the award and 
on performance. Another used the ad- 
ministrative portion for research, 
economic development planning, and 
workshops for PIC members. 

^ Eleven SDAs (37%) described spe- 
cial projects that they had funded 
with the 6% award. Several SDAs 
funded projects that targeted specific 
groups of hard-to-serve clients — 
operating projects such as dropout 
prevention, English-as-a-second-lan- 
guage programs, basic skills remedia- 
tion, a special project for youth with 
learning disabilities, and adult 
remedial education ("Get a GED, Get 
$500"). Projects such as these tended 
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to emerge in States where services 
funded with 6% funds were not in- 
cluded in the calculation of the perfor- 
mance standards.* 

• At least 15 SDAs (50%) had "folded 
in" the 6% aw rd so that it was vir- 
tually indistinguishable from their 
78% funding. 
Respondents described several other spe- 
cial ways in which they had used incentive 
funds. One SDA used 6% funds to reward 
contractors according to their performance, 
adjusted by the mode!. 

Incentive funds tended to be most impor- 
tant to SDAs that (1) had developed innova- 
tive special projects using those funds and 
had a strong interest in being able to continue 
those or similar projects, (2) had experienced 
particularly sharp declines in 78% funding as 
a result of falling unemployment rates, and 
(3) did not use performance-based contracts 
and therefore felt the pinch of the 15% limit 
on administrative funding. 

Other Factors Affecting SDA 
Decisions About Clients, 

Services, and Costs 

Although the performance-standards sys- 
tem is clearly an important influence on SDA 
planning, a number of other factors that also 
influence SDA actions were described during 
the site visits. Local factors beyond the 
SDA's control that influence their decisions 
about program design and client selection in- 
clude the local unemployment rate, labor 
market structures, and availability of service 
providers. State and Federal factors that in- 
fluence program design include the Federal 
youth expenditure requirement (as adjusted 
and monitored by the States), the requirement 
for equitable service to dropouts and AFDC 



recipients, and the limitation on the suppor- 
tive services available to clients. These fac- 
tors are described in the next section. 

Impact of Local Environmental 
Factors Not Under SDA Control 
Our local case studies revealed that the 
characteristics of the local economy were ex- 
tremely influential in shaping the SDAs' 
program-design decisions and led to some of 
the most difficult dilemmas that faced SDAs 
and program operators. 

Unemployment Rate. SDAs with low 
unemployment had difficulties recmiting 
clients because immediate minimum wage 
employment was often an attractive alterna- 
tive to JTPA participation. To attract clients 
in a booming economy, SDAs had to under- 
take large recmitment efforts and often found 
an applicant pool that consisted of individ- 
uals with an unstable work history, basic 
skills deficiencies, and multiple personal or 
family problems. These SDAs also had to 
revamp their service programs to meet the 
needs of the population of less job-ready 
clients. These adjustments to the service 
design included adding basic skills remedia- 
tion and counseling components, as well as 
offering more generous needs-based pay- 
ments and child care payments. 

Although nearly every SDA in the 
sample described having to adjust to a level 
of funding under JTPA that was far lower 
than its total CETA funding, the SDAs that 
were experiencing declining unemployment 
rates were most affected by declining fund- 
ing levels at the same time they were trying 
to meet the needs of more disadvantaged ap- 
plicants. For some SDAs, the level of JTPA 
funding had dropped by 50% since the begin- 
ning of JTPA, and allocations for PY 87 and 
PY 88 were continuing to drop sharply. A 



♦In a State thatreouiies 6% funds to be used for hard-to-serve clients, a percentage of one SDA's funds were to 
be earmarked for services to long-term welfare clients. Another was hoping to set up a program for homeless in- 
dividuals. A third funded a displaced-homemaker program and a "work and learn" program for dropouts. The fourth 
conducted pre-remediation for those who cannot otherwise succeed in their regular training program. 
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total of 16 of tnc 30 SDAs (53%) visited 
were in the process of substantial retrench- 
ment due to declining Title II- A funding. In 
others, the SDA kept the same program 
design but reduced the size and duration of 
individual projects and put pressure on con- 
tractors to reduce unit costs. 

SDAs often felt constrained by the declin- 
ing allocation to offer shorter term services 
so that their limited funds would be spread 
over as many participants as possible. At the 
same time, the requirements of the client 
pool increa?'*4, indicating a need for more en- 
riched supportive services and greater se- 
quencing of services to make clients 
employable. To resolve this dilemma, SDAs 
looked for ways to leverage JTPA funds with 
other funds, utilized contractors that had a 
proven track record of fulfilling their cost 
and entered-employment goals, and took a 
generally cautious attitude toward new 
program operators. 

In SDAs with high unemplovment rates, 
it was easy to recruit program clients and 
there was pressure to respond to the im- 
mediate needs of unemployed applicants 
with a fairiy clear set of retraining and rc- 
employment-oriented services. These clients 
tended to be easier to serve in that they had 
more recent work experience and faced 
fewer personal barriers to employment. Fur- 
thermore, because these clients had recent 
work experience, the SDAs felt less of a need 
to develop programs that included remedia- 
tion or world-of-work orientation. 

Labor Market Opportunities. The in- 
dustry and occupational mix of jobs available 
in the local economy also influenced the 
SDAs* design decisions. Generally, SDAs 
funded training that would prepare in- 
dividuals for jobs that existed in the local 
economy. Iii one instance, the PIC realized 
that the SDA needed to shift from training 
for the declining manufacturing sector to 
training for the growing service sector. 
Another PIC emphasized training in areas as- 



sociated with opportunities for economic 
development. A third SDA declared that OJT 
was an mappropriate service for its " fiill 
employment" labor maiket because 
employers did not require a subsidy to make 
them willmg to hire JTPA trainees. Several 
rural SDAs decided that local opportunities 
were too limited and offered training i-^ oc- 
cupations that were available m nearby cities. 

Service-Provider Availability. In most 
urban areas, SDAs could choose from a 
variety of potential service provides. In rural 
areas or smaller counties, however, SDAs 
had only a limited choice of classroom train- 
ing providers, which in turn limited the types 
and costs of training that could be provided. 
One case study SDA, for example, had to 
send participants 80 miles to the closest large 
town (located m another SDA) to provide 
some t>'pes of training. Clients who were un- 
able to travel or wanted training in other oc- 
cupational areas were thus effectively 
prevented from entering the program. Other 
SDA directors worried that some types of 
clients had not been well served by 
"mainstream" institutions (for example, the 
school system and the employment service) 
and wished that more CBOs were available 
to provide training in their areas. 

Impact of Other State and Federal 

Policies on Program Design 

and Client Selection 

Other than die performance standards, 
the State and Federal factors tfiat influence 
local program designs are (1) die require- 
ment that 40% of JTPA Tide II- A funds be 
expended on youth, (2) the legislative re- 
quirement for equitable services to dropouts 
and welfare recipients, and (3) die Federal 
limitation on the percentage of the JTPA al- 
location diat can be spent on supportive ser- 
vices and the lack of stipends. 

Youth Expenditure Requirement Four 
of the case study SDAs (13%) mentioned 
diat die youth expenditure requirement had 
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had a substantial effect on their local 
program design, causing them to expand their 
youth-oriented programs and increase the 
resources devoted to youth significantly 
beyond the level they would otherwise have 
chosen. Several SDAs found this requirement 
particularly frustrating because it was so dif- 
ficult to recruit youth in a tight labor market. 
In addition, several SDAs said that they had 
to enroll a large number of youth to reach the 
targeted expenditure rate because they were 
operating fairly inexpensive youth services. 
The influence of this factor is discussed more 
fully in Chapter IX. 

Equitable Service Requirements. 
Several SDAs changed their program design 
to be responsive to the Federal requirement 
to serve dropouts equitably, as interpreted 
and monitored by their State. (A number of 
other SDAs argued with their States that the 
expected levels of service to dropouts were 
too high and could not be reached without al- 
tering their program design.) One response to 
this requirement was to increase the amount 
of basic skills remediation and GED assis- 
tance available to out-of-school youth. 

Although the requirement for equitable 
services to dropouts was often mentioned as 
affecting SDA and service-provider behavior, 
the parallel requirement for equitable ser- 
vices to welfare recipients was not difficult to 
meet and thus did not have as large an effect 
on SDA behavior. Although SDAs viewed 
welfare recipients as hard to serve because of 
their poor employment histories, nonetheless 
SDAs did view welfare recipients as ap- 
propriate JTPA clients because they could 
rely on their welfare grants for income sup- 
port during training. For this group, the most 
difficult question was ncr how to recruit 
them, but how to overcome the work disin- 
centives inherent in the welfare system. The 
efforts of SDAs to serve these two groups of 
hard-to-serve clients are described more fully 
in Chapter DC. 



Lack of Stipends and Limit on Suppor- 
tive Services Spending. Several SDAs men- 
tioned having increased recruitment 
difficulties because of the inability to pay 
stipends under JTPA programs. The lack of 
stipends also influenced SDAs' decisions 
about the duration of training programs. 
Qients were more able to support themselves 
through shorter and more intense programs 
(for example, meeting all day for a shorter 
number of days rather than 4 hours a day tor 
a longer number of days). Because clients 
sometimes dropped out of longer training 
programs to take non-training-related jobs, 
some SDAs helped their participants find 
part-time jobs to support themselves during 
training. Other SDAs increased their em- 
phasis on OJT, which provides immediate in- 
come to participants. 

The Federal limitation on expenditures in 
supportive services governed the amount of 
needs-based payments SDAs could offer to 
their clients. Sample SDAs varied widely in 
the amount of allowances offered to clients, 
with the most common range being from $30 
to $60 per week for basic lunch/ transporta- 
tion payments. A few SDAs supplemented 
this with child care allowances, and SDAs 
also paid for books, supplies, and work 
clothes in a number of cases. Only one 
sample SDA had provided payments that 
would cover actual living expenses. General 
needs-based T^ayments were bare-bones al- 
lowances because SDAs preferred to spend 
as much of their limited funds as possible on 
training activities. 

Summary and Conclusions 

Several typologies of SDA responses to 
the performance standards help explain the 
variations in how the performance standards 
influenced clients, services, and costs within 
the case study sample. Client-oriented SDAs 
give priority to their client objectives and 
orient their program-design and management 
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decisions around how this will affect their 
client goals. Employer-oriented SD As give 
priority to designing and implementing a ser- 
vice mix that will train individuals for jobs in 
demand in the local labor market. Perfor- 
mance-oriented SDAs are very oriented 
toward meeting or exceeding the perfor- 
mance standards and are willing to com- 
promise their client and service goals to 
further their performance goals. 

Another related typology describes the 
way in which the performance standards in- 
fluence the SDAs' program-design and 
management decisions. Some SDAs are indif- 
ferent to the standards when they design their 
programs, either because they don't care if 
they meet the standards or because the stand- 
ards are so easy to meet. Other SDAs adopt 
risk avoidance strategies that increase the 
likelihood that they will meet the perfor- 
mance standards by avoiding hard-to-serve 
clients or adjusting the service mix for 
reasons that are standards oriented. Finally, 
some SDAs adopt risk management 
strategies that enable them to meet their per- 
formance goals while still furthering local 
client and service objectives. 

A final typology describes how SDAs 
vary in the extent to which the SDA keeps 
responsibility for meeting its performance 
goals, shares that responsibility with service 
providers, or passes on the primary respon- 
sibility for performance to the service- 
provider level. 

The performance standards played an im- 
portant role in some SDAs, a minor role in 
others. Only a few SDAs in the case study 
sample were purely **risk avoidance" rather 
than "risk management" in their response to 
the performance standards. Most SDAs could 
meet the standards at the desired level with- 
out having to resort to a major revision of 
other program goals. 



The relative importance of the perfor- 
mance standards in shaping local goals and 
priorities was conditioned ^n part by the im- 
portance to the SDA of receiving the maxi- 
mum amount of incentive funds. Some SDAs 
placed high priority on receiving funds be- 
cause they viewed them as compensating for 
shrinking 78% allocations and tight admini- 
strative limits. Others valued the awards for 
their public relations value. Still others used 
the awards to undertake projects that they 
could not fund with 78% money, such as 
services for high-risk clients, or marketing or 
outreach projects that only indirecdy contrib- 
uted to outcomes. 

Other local. State, and Federal factors in- 
fluenced me SDAs* program- design and 
client selection priorities as much or more 
than the performance standards. These in- 
cluded local environmental factors beyond 
the SDAs* control, such as (1) the local un- 
employment rate, which affected the com- 
position of the available client pool; (2) the 
local labor market opportunities, which af- 
fected the type of training offered; and (3) 
service provider availability, which also af- 
fected the types of training that could be of- 
fered. The State and Federal factors that 
influenced the SDAs' behavior included the 
40% youth expenditure requirement and the 
equitable service requirements for dropouts 
and welfare recipients, both of which in- 
fluenced SDAs to target groups that they 
may not have otherwise done. Another factor 
was the limit on supportive service costs and 
lack of stipends, which caused recmitment 
difficulties for SDAs and required changes in 
service duration and intensity to ensure that 
clients could support themselves through the 
training program. Often these other factors 
influenced the SDAs' program-design 
decisions as much or more than did perfor- 
mance standards. 
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Introduction 

The Federal performance-standards sys- 
tem was designed to avoid influencing local 
decisions about clients served. Aside from 
the requirements that 90% of JTPA recipients 
be economically disadvantaged, that 40% 
of the funds be spent on youth, and that 
dropouts and welfare recipients be equitably 
served, the Federal Government leaves 
decisions about whom to serve to the States 
and local SDAs. The qualitative analysis ex- 
amined whether the existence of the Federal 
performance-standards system had unin- 
tended effects on who was served under 
JTPA programs. The qualitative analysis also 
examined how State performance-standards 
policies affected the SD A*s choices about the 
types of clients served. To answer these ques- 
tions, we conducted interviews with SDA, 
PIC, and service-provider staff to determine 
the overall client goals as well as outreach 



and recruitment practices used in the case 
study SDAs. These discussions explored the 
factors that affect the SDAs* client goals and 
practices and whether the performance stand- 
ards were preventing the SDAs from enroll- 
ing client groups that had been identified as 
priorities in the local area. 

Overview of Chapter 
and Key Findings 

The first part of this chapter examines the 
factors that influence the SDAs' goals for 
clients serv ed. Two-thirds of the SDAs 
visited identified priority client groups, either 
in their plans or by establishing special proce- 
dures or programs. Overall, Federal perfor- 
mance-standards policies had little effect on 
the SDAs* client goals. State policies and 
local factors had greater influence. In States 
that had policies for equitable service 
provision or incentive policies that em- 
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phasized service to hard-to-serve clients, 
SDAs often reflected these State priorities in 
their local client goals. In addition, youth ex- 
penditure requirements and the characteris- 
tics of the applicant pool also influenced the 
SDA's client goals. 

The second part of the chapter examines 
barriers to serving hard-to-serve groups and 
the strategies some SDAs used to overcome 
those barriers . Welfare recipients, dropouts, 
handicapped, offenders, and those with 
limited English skills were identified as hard 
to serve by many SDAs. Strategies for serv- 
ing these groups included designing special 
services for their needs, implementing ser- 
vice strategies or employer-targeting 
strategies, leveraging JTPA funds with other 
sources, and making explicit tradeoffs be- 
tween short-term services to better prepared 
clients and long-term services to clients with 
greater needs. 

SDA Client Goals 

Of the SDAs visited, two-thirds (20 
SDAs) had developed a set of local client 
goals and priorities for service to particular 
subgroups of the eligible population. Very 
often these built on State goals and policies, 
but they usually went beyond to target other 
subgroups. Of those SDAs with specific 
goals, nine (30%) had a stated commitment 
to serving hard-to-serve clients within Title 
II-A programs, and four SDAs (13%) ex- 
pressed a clear policy of serving a broad mix 
of clients, including a mix of less job-ready 
and more job-ready individuals. 

SDAs implemented client goals in a num- 
ber of ways: by declaring groups to be high 
priority, by conducting special outreach, by 
developing priority systems for enrolling 
clients, and by funding programs for par- 
ticular groups. 

SDAs usually listed priority groups in 
their plans, usually incorporating State goals 



for service to welfare recipients and dropouts 
and the required youth expenditure goal. 
However, SDAs also designated other 
populations as target groups. Those men- 
tioned most often were high-risk or multi- 
problem youvh, handicapped, minorities, 
offenders, women, limited-English speakers, 
single heads of households, Vietnam 
veterans, and older workers. A rural SDA tar- 
geted dislocated farmers, and a few multi- 
county SDAs had pclxv^ics about geograph- 
ical equity. Sometimes the service-level 
goals listed in the plan got translated into 
specific practices or programs, but more 
often they were only used as planning figures 
against which enrollment numbers were 
checked throughout the year. 

About one-third of the SDAs had in- 
stituted practices for special outreach to 
recruit members of the target groups. Out- 
reach practices included running ads on eth- 
nic radio stations or in local papers; 
enclosing special brochures in welfare check 
envelopes: making presentations at com- 
munity group meetings; and sending letters 
to school dropouts. 

Three SDAs (10%) had many more ap- 
plicants than they could serve and therefore 
had developed point systems for prioritizing 
eligible applicants. Usually applicants with 
more barriers (for example, long-term female 
welfare recipients with little work ex- 
perience) received more points than those 
with fewer barriers (for example, recently un- 
employed individuals). This practice allowed 
SDAs to target their services to those most in 
need. However, when the economy was heal- 
thy and clients were hard to find, priority sys- 
tems and targeted groups were generally left 
behind as SDAs scrambled to find enough 
eligible uidividuals to fill the available slots. 

Another conimon way of targeting some 
client groups was to fund programs specifi- 
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cally for them. This was a practice used most 
often for the handicapped and offenders.* 
Nine of the case study SDAs (30%) had 
separate programs for these groups, and three 
more (10%) targeted them for special out- 
reach. Because these groups were seen as 
having special needs, the SDAs usually con- 
tracted with organizations that had ex- 
perience working with the handicapped or 
offenders. The specialized organization, 
which usually had funding from a number of 
sources, then took responsibility for the full 
range of services, from recruitment to place- 
ment and follow-up. 

A few other subgroups received special 
attention in a less consistent way: one SDA 
ran a special program to train women in non- 
traditional occupations, another had one for 
recovering alcoholics, and several SDAs 
awarded contracts to CBOs that offered ser- 
vices to specific ethnic communities (for ex- 
ample, Hispanic, Portuguese, and Samoans). 
Some SDAs ran separate programs for wel- 
fare recipients by using money the State 
made available for that purpose or using the 
State money to allow welfare clients to 
receive more services than other clients (for 
example, 8 weeks of basic skills remediation 
instead of 4). 

Ten of the SDAs visited (33%) had no ex- 
plicit local client goals. They either adopted 
the State goals without modification or said 
that their goals were to serve "all the disad- 
vantaged" or "any eligible client." These 
SDAs included a number of large central 
cities where staff reported that "anyone walk- 
ing in the door is hard to serve" and thus did 
not feel a need for elaborate statements of 
goals and priorities. Some SDAs viewed 
priority groups as "mere politics." "We're 
tired of bemg told we should serve the latest 
trendy group — the latest was teenage 
fathers," reported one SDA director. Another 



asked, "Can you believe that the governor 
thinks we should design a program for the 
homeless?" (On the other hand, another SDA 
wanted to develop such a program using its 
6% award.) Some SDAs explicitly decided 
against offering special programs for specific 
subgroups, on the grounds that clients 
learned more in mixed groups and that they 
were stigmatized by being served separately. 

Among the SDAs without client priorities 
were a number of large multicounty SDAs. 
These SDAs divided up the allocation and 
passed funds through to the sub-SDA coun- 
ties, each of which designed its own program 
and set its own client goals. Because the 
needs of the eligible population might vary 
greatly from county to county, these SDAs 
felt it was more appropriate to take a neutral 
role with respect to client priorities. 

Influence of State Policies 

on SDA Client Goals 

States influenced the SDAs' client goals 
through several practices: modifying incen- 
tive policies to encourage service to the hard 
to serve, monitoring equitable service goals, 
and supplementing JTPA funding with other 
funds to increase service to welfare recipients. 

None of the States visited had instituted 
explicit additional performance standards for 
specific client groups in PY 86. However, 
two States had put client priority standards in 
place since then. One State had instituted an 
optional positive-termination-rate standard 
for high-risk youth in PY 87. Another State 
was planning to implement a standard for ser- 
vice levels to welfare recipients in PY 88. 
The anticipation of future requirements made 
SDAs in that Stale emphasize welfare 
recipients even before the standard was in- 
stituted. 

Two States encouraged service to par- 
ticular groups by awarding substantial 
amounts of 6% incentive funds based on ser- 



♦Among the handicapped, the subgroups usually targeted were the menially ill or the mentally retarded, and of- 
fender programs sometimes differentiated between adults and youth. 
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vice to targeted groups. One State used 30% 
of the 6% funds to reward service levels to 
minorities and dropouts; the other State 
weighted service levels to AFDC recipients 
and dropouts as heavily as overall perfor- 
mance in the incentive formula in PY 87. 
Even though these policies were not iden- 
tified as separate performance standards, 
most SDAs had a goal of maximizing incen- 
tive funds and therefore changed their prac- 
tices to conduct special outreach and recruit- 
ment for clients with those characteristics. 

Although these policies influenced SDA 
behavior, often the SDAs disagreed with the 
client priorities expressed by the State incen- 
tive policies. In both States described above, 
the goals for services to dropouts were 
generally seen as particularly onerous be- 
cause of the high level at which service ex- 
pectations were set, which SDAs often felt 
were unreasonable and unattainable goals. 
(The States used census figures for the per- 
centage of dropouts in the general population 
as goals, and SDAs felt these goals were 
higher than the percentage of dropouts in the 
applicant pool.) However, SDAs generally 
agreed with the emphasis on welfare 
recipients because the incidence of welfare 
recipients in the population was easier to 
compute and recruitment was less of a 
problem for this group. 

States also attempted to influence the 
level of service to welfare recipients by heavi- 
ly weighting the welfare entered-employ- 
ment-rate standard in their incentive policif s. 
Consistent with the results of the quantitative 
analysis, in the case study sites this policy 
lad less of an influence on the number of 
clients served than policies about the level of 
service, because the rate of employment can 
be high on a small absolute number of 
clients. However, several SDAs reported that 
the high weight on this standard made them 
pay more attention to the group than they 
otherwise would have. 

All of the case study States monitored ser- 
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vice levels to various groups, even when they 
did not attach incentive funds to meeting the 
service goals. The SDAs generally tried to 
comply with the goal of service to the 
monitored groups because doing so made the 
plan approval process go more smoothly. 
The groups usually monitored were those 
mentioned in the JTPA legislation: AFDC 
recipients and dropouts. One State went 
beyond these groups and monitored service 
to women, minorities, adults over 40, and the 
handicapped, and another monitored service 
levels to all the groups listed on the JASR, 
plus additional ethnic categories. 

States also communicated client goals 
through alternative funding streams. Six 
States had instituted work- welfare programs, 
and three awarded State and Federal funds to 
SDAs to serve this group. Because the States 
with strong work-welfare programs wanted 
to coordinate with the existing JTPA training 
system, they reasoned that offering SDAs 
funding to serve those clients was the best 
way to ensure that the welfare recipients 
would be served. 

These results of the case studies are very 
consistent with the results of the quantitative 
analysis presented earlier. State policies that 
encourage service to welfare recipients, 
dropouts, and those with other barriers to 
employment were found to significantly in- 
crease service to these groups. In the case 
studies we found that many SDAs included 
State priorities in their own goals for th 5 
types of clients served in JTPA programs. 

Influence of the Youth 
Expenditure Requirement 

Youth became a priority group in many 
SDAs by virtue of the requirement that 40% 
of expenditures (as adjusted for actual SDA 
characteristics) be spent on them. Although 
no State had attached any incentive money to 
the fulfillment of the requirement, youth ex- 
penditure rates were monitored carefully. 

The majority of SDAs believed that this 
de facto emphasis on youth was appropriate 
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because they felt it was important to prevent 
employment problems early in life by keep- 
ing youth in school and giving them training 
and work experience. Three SDAs (10%) tar- 
geted even more money than was required on 
youth. However, at least four SDAs (13%) 
felt that the youtli expenditure requirement 
prevented them from servmg adults who 
needed and wanted help more; for these 
SDAs, the requirement was an imposition 
on local J^'nonomy. 

Whether they agreed with the policy em- 
phasis or not, many SDAs had trouble meet- 
ing the youth expenditure requirement 
because they had difficulties recruiting 
enough youth. SDA staff had a number of ex- 
planations for this difficulty. Some SDAs 
stated that youth were not future oriented: 
"Youth would rather make minimum wage 
flipping burgers than stay in training for the 
chance at a higher paying job." This problem 
was especially acute in areas with booming 
economies, where even fast food jobs paid 
above the minimum wage. Others cited a 
lack of motivation for youth: '*They don*t 
have family responsibilities, and theyVe get- 
ring by — by livmg with their families or deal- 
ing drugs or crime. TheyVe not going to 
spend time in training programs." One 
frustrated SDA director claimed that "youth 
are an invisible population." 

SDAs often went to great lengths to en- 
roll a sufficient number of youth, especially 
out-of-school youth. Strategies included 
hiring young people as recruiters to knock on 
doors and visit basketball courts, offering 
bounty payments to community groups for 
successful referrals, and streamlining 
eligibility processes so that youth were en- 
rolled more quickly than adults. SDAs also 
passed on the youth expenditure requirement 
to their contractors by including quotas in 
contracts. 

Characteristics of the Applicant Pool 

The local environment had an influence 
on the characteristics of the applicant pool. 



which in turn influf ;nced who was served by 
the SDAs. In the 16 SDAs (53%) with low 
unemployment rates and booming 
economies, the remaining unemployed in- 
dividuals were likely to be those with multi- 
ple barriers to employment. Most of these 
SDAs were shifting from serving a client 
pool dominated by dislocated woikers or at 
least clients with some work experience to a 
client pool composed mostly of long-term 
welfare recipients, individuals with emotion- 
al problems and histories of substance abuse, 
and persons with minimal and intermittent at- 
tachment to the work force. SDAs that had 
not previously served such a large number of 
these hard-to-serve clients were making ad- 
justments in their assessment procedures and 
service designs to accommodate the new ap- 
plicant pool. 

In the remaining 14 SDAs (47%), in 
which unemployment remained high and the 
economy was contracting or shifting from 
manufacturing to service jobs, there was less 
of a need to address multiple barriers to 
employment because many clients had recent 
work experience and were not perceived as 
hard to sen/e. In these cases, SDAs con- 
centrated on the retraining needs of in- 
dividuals. 

Again, the quantitative analysis substan- 
tiates these results. The statistical analysis in- 
dicates that areas with low unemployment 
rates serve significantly more dropouts, 
minorities, and those with other barriers to 
employment. 

Hard-to-Serve Groups: 
Barriers and Strategies 

There has been considerable debate about 
whether the performance standards instituted 
under JTPA programs have prevented SDAs 
from serving hard-to-serve client groups. Al- 
though there is no one definition of hard to 
serve, the groups most often mentioned are 
welfare recipients and dropouts; these are 
also the groups for which equitable service is 
mandated. Other barriers to employment that 
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are neasured on the JASR are being an of- 
fender, having a disability, and having 
limited English skills. 

These client subgroups are the ones for 
which we have statistics and are discussed in 
detail in a later section. Many of the SDAs 
visited, however, pointed out that the JASR 
variables often do not fully capture the hani- 
to-serve nature of their clients. For instance, 
the mere fact of being a welfare recipient 
does not necessarily make a client difficult to 
serve. However, a client comes closer to 
meeting the definition of hard to serve when 
she is a long-term welfare recipient who 
dropped out of high school, with no work his- 
tory and a need for health benefits and child 
care. Often a combination of the characteris- 
tics described above made clients hard to 
serve or other factors inherently difficult to 
measure, such as substance abuse problems, 
mental illness or other mental health 
problems, a history of job hopping or other 
indications of nonattachment to the work 
force, 01 other motivation problems. 

Another major unmeasured factor that 
makes a client hard to serve is the lack of 
basic skills. In some SDAs, the inability to 
read or perform simple calculations was a 
serious barrier to training, let alone employ- 
ment, for a number of clients. One assess- 
ment provider pointed out the extent of the 
problem of basic skills deficiencies: "People 
in Washington have no idea of the depth of 
ignorance that exists in cities. Until you've 
worked v/ith these people — and find out that 
they really can't tell time, or that they've 
never learned measurement concepts like in- 
ches or quarts — you have no concept of what 
it means to get the truly hard to serve ready 
for training. According to our contract we 
should not refer them for training unless they 
can be ready for placement within three 
months. Without remedial work, how can 
you possibly avoid taking the cream of the 
crop?" 

If a client with low levels of basic skills 



had a previous work history and did not 
desire classroom training, the staff of most 
SDAs would be able to find the client an 
OJT position similar to his or her old job. 
But for clients without a work history, the 
lack of basic skills often had to be addressed 
before classroom training could be com- 
pleted. 

Some classroom training corricula in- 
cluded both basic skills and occupational 
skills training, on the assumption that clients 
would more readily learn if they could see 
how skills were applied. However, in other 
SDAs, clients had to obtain basic skills train- 
ing outside of the JTPA system before return- 
ing for occupational training. SDAs gen- 
erally did not keep track of the number of ap- 
plicants who were referred to adult basic 
education or other sources of basic skills 
training and never returned to JTPA 
programs, but this was an acknowledged 
source of clients "dropping through the 
cracks." 

SDA staff sometimes perceived that per- 
formance standards prevented them from 
serving more hard-to-serve clients, becau'^e 
they thought that the cost standards prevent- 
ed them from offering the long-term pro- 
grams that low-skilled clients would need to 
make them employable. Furthermore, even 
when they were willing to spend larger 
amounts of money on their worse-off clients, 
some SDAs worried that they would only be 
able to prepare them for low-wage jobs and 
would thus endanger their performance level 
on the wage standard. However, in fact, few 
of the SDAs visited had missed any stand- 
ards, and in most cases, they were perform- 
ing well above their required performance 
levels. 

However, other factors not related to per- 
formance standards also weighed heavily on 
the minds of SDA staff. One was the neces- 
sity to meet the needs of the employer com- 
munity and thus to have a range of clients to 
offer for employment. "We don't want to go 
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to employers and say *just hire the downtrod- 
den/ " said one SDA director. "We want to 
offer them more experienced people as well 
as entry-level people." Another factor that 
was very important to several SDAs was the 
desire to spread limited JTPA funds over as 
many clients as possible, which led to an ex- 
plicit policy uf serving a less hard-to-serve 
range of clients who could benefit from a fair- 
ly short-term program. 

However, what is striking is that a con- 
siderable number of case study sites did tar- 
get and serve what would be considered 
hard-to-serve clients. Although the staff at 
these sites acknowledged that barriers ex- 
isted, they often found creative strategies for 
overcoming those barriers and addressing a 
wide range of client needs. The sections 
below discuss the strategies used for the most 
commonly cited hard-to-serve groups. 

Welfare Recipients 

A group that was often perceived as par- 
ticularly hard to serve was long-term welfare 
recipients. Barriers to serving this group con- 
sisted of both the clients* limited employ- 
ment readiness and work disincentives 
created by the welfare system. Welfare 
clients generally had low levels of basic 
skills and spotty work histories that made it 
difficult to train them for high-wage jobs in 
short-term programs. Work disincentives 
were built into the welfare system. Recipients 
(especially in high-benefit States) were often 
better off on welfare than in a low- wage job 
because of the loss of cash grant and, more 
importantly, the loss of health care benefits. 
In addition, welfare clients usually incurred 
child care expenses as well as the normal 
work expenses associated with taking a job. 
Thus, it was necessary to place welfare 
recipients in high-paying jobs with health 
benefits to make them as well off as they had 
been on welfare. 

In spite of these barriers, welfare 
recipients were quite well represented among 
the terminees in the case study States. Work- 



welfare requirements as well as State incen- 
tive policies that encouraged service to wel- 
fare recipients made SDAs more willing to 
serve this group. However, State priorities do 
not fully explain the presence of so many wel- 
fare recipients in JTPA programs. Many 
SDA staff reported that welfare recipients are 
especially attractive clients because they are 
least likely to be deterred from obtaining 
JTPA services due to the lack of stipends. Al- 
though the stipends offered under CETA 
were often maligned as causing clients to ob- 
tain training for the sake of training rather 
than for the sake of employment, they also 
enabled clients with no other possibility of in- 
come support to complete training programs. 
Nonwelfare clients who were only receiving 
at best a $20-60 per week transportation al- 
lowance were known to quit JTPA training 
programs in order to take jobs. Welfare 
recipients, on the other hand, were sought out 
by SDAs because of their higher probability 
of completing a training program. 

A few SDAs had developed strategies for 
serving welfare recipients. One SDA had 
made it a policy to refer these clients only to 
jobs that paid $7.00 per hour or above, which 
was the "break even" point at which they 
would be as well off as on welfare. However, 
this strategy requires the SDA to make a 
large investment in training, usually taking at 
least a year and including basic skills 
development, occupational skills training, 
and perhaps work experience or an OJT posi- 
tion. Another SDA made the transition easier 
for welfare mothers by buying them a health 
insurance plan for a year after they were 
placed in a job. This is also an expensive op- 
tion, costing about $2500 per person. A third 
SDA made it a policy to write OJT contracts 
(for all participants, not just welfare 
recipients) only with companies that offer 
health benefits. 

Dropouts 

A number of SDAs expressed frustration 
at their inability to offer more adult remedia- 
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tion because there was no recognition of a 
positive outcome other than employment for 
adults. These SDAs felt that the eligible 
population contained many people who 
needed a GED before they would be able to 
progress to a decent job and were interested 
in offering this service. However, SDAs real- 
ized that tliey could get no credit for a "GED 
only" outcome and that the remediation 
presently had to be built into a sequence of 
services that included skills training or job 
placement assistance. If the client needed 
skills training after the remediation, the 
program became both long (leading to a high 
program dropout rate) and expensive (caus- 
ing SDAs to be nervous about the cost stand- 
ard). In other cases, the client simply attained 
the GED and did not go on to a job, which 
meant a negative termination for the SDA. 
Some SDAs made explicit decisions not to 
target adult dropouts, reasoning that they 
could not be "all things to all people." 

However, because States are required to 
monitor equitable service levels to this group 
and because States sometimes go beyond that 
requirement to declare them a priority group, 
SDAs are often forced to take action if 
dropouts do not make up a large percentage 
of the applicant pool. 

The tv/o case study SDAs with the 
highest level of service to dropouts had ex- 
plicit PIC policies to offer services to the 
hardest to serve. SDA staff reported that 
these local priorities, rather than State 
policies, influenced their level of service to 
dropouts. On the opposite side, neither of the 
SDAs with the lowest levels of service had 
policies or programs to reach out to dropouts 
in PY 86, although one of them had since in- 
stituted a special $500 bonus to adult 
dropouts to encourage them to get their 
GEDs, paid for out of 6% funds. 

SDAs used a number of strategies to 
offer remediation services to allow dropouts 
to obtain their GEDs. SDAs that felt the 
standards prevented them from offering adult 
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remediation sometimes provided this service 
using 8% funds; others took advantage of 
State education funds available for GED in- 
struction. The SDAs that made remediation 
available as part of their Title H- A program 
often utilized nontiaditional methods of in- 
struction, such as self-paced computer-as- 
sisted study before or concurrently with 
occupational skills training. Scaff felt that 
this approach increased the chances that their 
clients would succeed, often at a faster pace 
than traditional teaching allows. 

One SDA suggested that SDAs be al- 
lowed to offer "remediation only" to adults 
who did not then go on to immediate employ- 
ment and to take those clients out of the 
denominator when calculating performance 
on the adult entered-employment standard. 
SDAs who were interested in this strategy 
were willing to have a cap on the number of 
adults who could be served this way. One ser- 
vice provider pointed out that for the pur- 
poses of obtaining basic skills, the distinction 
between a 21-year-old "youth" and a 22-year- 
old "adult" was purely arbitrary. 

Other Barriers to Employment 

Other categories of clients who are often 
considered hard to serve are offenders, hand- 
icapped, and those with limited English 
skills. These groups are all perceived to have 
special needs, either for training (hand- 
icapped, limited English) or placement (of- 
fenders) or both (handicapped). However, 
several SDAs in the case study sample tar- 
geted one or more of these groups. 

The most common strategies for serving 
these groups was to fund separate programs 
for them. By contracting with CBOs or- 
ganized to serve the targeted group, the 
SDAs accomplished two things: they got a 
service program especially designed for the 
needs of the group, and they could buy the 
services below cost because the organiza- 
tions usually had other funding sources that 
allowed them to offer an enriched set of ser- 
vices. Sometimes agencies that offered social 
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services or language instruction could use 
JTPA funds to "add on" skills training or 
placement assistance for their most job-ready 
clients, thus offering the SDAs good out- 
comes at a fairly low cost. In other SDAs, the 
outcomes for these special needs populations 
were lower than for other JTPA clients, but 
the SDAs had a commitment to serving the 
group. 

General Strategies for Serving 
the Hard to Serve 

Many SDAs perceived cost constraints to 
be an important impediment to serving hard- 
to-serve clients. However, the available data 
show that most SDAs were in no danger of 
missing die cost standards, even if they in- 
creased dieir costs by 50 or 100%. Several 
factors odier than the fear of missing die 
standard seemed to be at work here. One was 
the desire to exceed the standard as much as 
possible to maximize die 6% award received 
This is consistent with die quantitative results 
that SDAs in States with incentive policies 
that emphasize exceeding standards are less 
likely to serve several hard-to-serve groups. 
The otiier factor that inhibited some SDAs 
from serving more hard-to-serve clients was 
the political desire to spread the available 
funds over as many clients as possible. 

Nonetheless, many SDAs developed 
several strategies for serving hard-to-serve 
clients at low cost. The most common 
strategy was to use funds from other sources 
to supplement scarce JTPA funds. Thus, 
SDAs commonly contracted with local 
school systems, community colleges, or 
CBOs to provide classroom training. By con- 
tracting with a local vocational school or 
community college, die SDA could usually 
buy a longer service program for a lower 
price. In addition, public educational or- 
ganizations and CBOs were usually more 



willing to develop classes especially suited 
for die hard to serve among die JTPA popula- 
tion, whereas the proprietary institutions 
were more likely to accept individual par- 
ticipants on a "slot in" basis. The existence of 
a class-size program meant diat basic skills 
remediation and world-of-work orientation 
could be included in die curriculum radier 
dian as separate services. Offering all the ser- 
vices togetiier was seen as important for meet- 
ing die needs of multiple-problem clients.* 

Odier sources of additional funding in- 
cluded State, local, and foundation funding. 
One State visited provided State funds for 
supportive services for JTPA participants so 
diat SDAs could use more of their Federal 
funds for training. SDAs in that State 
reported that die State supportive service al- 
location generally lasted for duee quarters of 
the year. Some States offered additional fund- 
ing to serve a specific group, such as welfare 
recipierts or high-risk youth, so that services 
could bo expanded. One SDA had obtained 
some foundation funding but reported that it 
came with nc administrative funds and that 
the accounting for separate streams was more 
trouble than it was worth. 

By leveraging JTPA funds with non- 
JTPA funds, SDAs could offer longer term 
services to their hard-to-serve clients and 
meet dieir performance goals. However, in 
diese SDAs, die JTPA costs reported on die 
JASR in many cases understate the true 
amount of money being spent on services to 
JTPA clients. 

As we discuss in detail in Chapter X, 
anodier way SDAs addressed dieir cost goal ? 
and offered sufficient services to haid-to- 
serve clients was to make explicit tradeoffs 
between short-term services to better 
prepared clients and long-term services to 
clients with more needs. For instance, one 



♦However, for many sparsely populated SDAs. class-size training is impractical, both because it is too difficult 
to assemble a class from a geographically dispersed area and because the economy will not absorb a class of 
graduates atone time. 
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SDA offered job-search assistance to a wide 
range of clients while limiting its classroom 
training to those with little work experience. 
Another offered short-term OJT to clients 
with work experience to offset the cost of ex- 
pensive classroom training for less prepared 
clients. These strategies were also consistent 
with SDAs* desires to offer the employer 
community a wide range of potential 
employees. 

Strategies for serving those most in need 
while still meeting their wage-rate goals were 
harder to come by. Often, this was the hard- 
est of the adult standards to meet, and some 
SDAs exceeded it by only a few cents. The 
most common strategy for maximizing the 
outcome on this standard was to limit train- 
ing to occupational areas with a high propor- 
tion of higher paying jobs. Other SDAs made 
explicit tradeoffs by acknowledging that 
some clients could only get entry-level jobs 
and offsetting them with clients who could 
get jobs at wages considerably above the 
standard. For instance, clients with prior 
work histories who attained higher paying 
jobs in areas such as drafting or machine 
work could b^^^ance less job-ready clients 
who attained lower paying clerical jobs. 

Another strategy for getting better wages 
for hard-to-serve clients was to write 
"stepped" OJT contracts with employers. 
Under these contracts, the client started at 
minimum wage or just rbove and proceeded 
through wage increases until a wage at or 
above the standard was reached. SDAs 
reported that such contracts were more attrac- 
tive than an all-or-nothing demand for high 
starting wages for clients about whom the 
employer had reservations. 

Another SDA practice that improved ser- 
vice to the hard to serve was extensive assess- 
ment and orientation sessions, which helped 
direct clients to appropriate training. This 
could take the form of a vocational assess- 
ment by an outside vendor (for example. 
Goodwill), tests administered by the SDA, or 



"try out" sessions in which clients sat in on 
classes that they were nost interested in. 
Often these try out sessions served as reality 
tests for clients who had overinflated goals 
and helped to direct them to the occupational 
field in which they were most likely to suc- 
ceed. 

Summary and Conclusions 

Performance standards had only a limited 
impact on the SDAs* client goals. Local con- 
cerns and PIC priorities for targeting par- 
ticula'. client groups were more likely than 
the performance standards to lead to special 
outreach and recruitment or specially 
designed programs for hard-to-serve clients. 

Other factors that influenced client goals 
were State equitable service requirements 
and client priorities, the 40% youth expendi- 
ture requirement, and the characteristics of 
the applicant pool, which were strongly in- 
fluenced by environmental factors beyond 
the SDAs* control, such as the unemploy- 
ment rate. 

The results of the case studies are consis- 
tent with the quantitative analysis, which in- 
dicates that State policies for serving client 
groups can increase service to those groups. 
Furthermore, the weight on the welfare 
entered-employment rate did not have a 
strong influence on service to welfare 
recipients. 

The performance standards did not 
prevent many SDAs from targeting and serv- 
ing hard-to-serve groups. Strategies for serv- 
ing these clients included 

• Designing special training classes 
that integrated remediation and world- 
of-v/ork orientation with occupation 
skills training, 

• Leveraging Title II-A funds with 
other JTPA funds (for example, 8% 
or 6%) or non-JTPA funds (for ex- 
ample. State funds or foundation 
funds). 
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Targeting employers who offered 
health benefits, 

Making explicit tradeoffs between 
providing hard-to-serve clients with 
high-cost services and providing 
more job-ready clients with short- 
term services, and 

Funding special programs operated 




by providers with a history of work- 
ing with clients with other barriers to 
employment (handicapped, offenders, 
and limited-English speakers). 
The strategies we observed in the case 
study sites are primarily risk-managing 
strategies that allowed SDAs to attain their 
performance goals and to meet their client 
goals. 



X. Impact of Performance Standards 
on Service Mix, Intensity, and Costs: 
Evidence from the Qualitative Analysis 



Introduction 

The JTPA legislation leaves the choice of 
service design to the discretion of the local 
SDAs and PICs. Aside from a prohibition of 
public service employment as a service, the 
legislation mandates no required service mix 
or duration. Rather, SDAs, in conjunction 
with their local PICs, are to choose the mix 
of services that best meet local needs. Federal 
performance-standards policies were in- 
tended to lead SDAs to choose cost-effective 
services but not to constrain unduly the 
SDAs' ability to offer services appropriate 
to local needs. The qualitative analysis ex- 
amined whether the Federal performance- 
standards system had unintended effects on 
the kinds of services SDAs decided to offer. 

Overview of Chapter 
and Key Findings 

This chapter traces the effects of the per- 



formance standards on several dimensions of 
program-service design: 

• Service mix . The sen/ice mix is the 
amount of occupational classroom 
training, OJT, basic skills remedia- 
tion, and direct job-search assistance 
offered with Title II- A funds. We 
found that performance standards had 
only a modest influence on the ser- 
vice mix offered to adults by case 
study SDAs. The historical pattern of 
service delivery, kinds of clients the 
SDAs wanted to serve, and PIC 
preferences about service design were 
greater influences than performance 
standards. 

• The intensity and duration of ser- 
vices . The cost standard had some in- 
fluence on the intensity and duration 
of service because SDAs often 
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wanted to not only meet but to ex- 
ceed this standard to maximize their 
6% awards. Other factors that were 
equally as important in determining 
the intensity and duration of services 
were the availability of non-JTPA 
funds, the costs of purchasing ser- 
vices from local service providers, 
and the SDA's goals about the type 
and number of clients to serve. 

• The orientation of youth programs . 
There were variations in the types of 
specialized youth programs offered 
(for example, for out-of-school ver- 
sus in-school youth). The existence of 
performance standards for youth did 
not seem to affect the extent of SDA 
resources devoted to youth programs 
per se. However, SDAs that offered 
competency-oriented programs often 
found it difficult to meet their entered- 
employment-rate standard, and SDAs 
offering employment-oriented 
programs often found it difficult to 
meet their positive-termination-rate 
standard. 

The final section of this chapter examines 
JTPA sen/ice costs. 

• The costs of JTPA services . There 
were variations in the range and 
average costs of serving JTPA par- 
ticipants within and across SDAs. 
The unit costs of providing JTPA ser- 
vices varied dramatically among 
SDAs, and this aspect of SDA perfor- 
mance was the least uniformly 
measured across SDAs due to the 
wide variation in the extent to which 



SDAs relied on non-JTPA funds. 
SDAs v/ere very concerned about 
reducing j f PA costs to reach as 
many individuals as possible with 
limited funds and to exceed their cost 
standards. 

Service Mix 

Among the major SDA design decisions 
about program mix were the following: (1) 
whether to focus on occupational skills train- 
ing (for example, classroom training in par- 
ticular occupational skills or OJT) with the 
Title n-A funds or to broaden the focus of 
Title II-A services to include major expendi- 
tures for remediation in basic skills and/or 
direct job-search assistance and (2) whether 
to emphasize OJT or classroom training or a 
mix of both within the occupational skills 
training area. 

An Emphasis on Occupational 
Skills Training or a Broader 
Range of Services 

One interesting outcome in analyzing the 
service packages offered by SDAs is the ex- 
tent to which Title II-A funds were used to 
provide occupational skills training as op- 
posed to other activities. The two categories 
of occupational skills training offered in the 
case study SDAs were occupational class- 
room training in occupational skills and OJT. 
For adults, the two activities that do not fall 
into the occupational skills training category 
are basic skills remediation and direct job- 
placement/job-search assistance.* 

The extent to which JTPA Title II-A ac- 
tivities were oriented toward occupational 
skills training versus other activities varied 
substantially among the 30 case study SDAs. 



*The information that we obtained from SDAs included information about the number of terminees or par- 
ticipants receiving each of these categories of service. In general, for basic skills remediation and job-search assis- 
tance, we received information about the number of individuals who had been enrolled in this activity as a discreet 
"stand-alone" service, rather than the total number for whom basic skills remediation or job-search assistance v/as 
provided as an integrated part of the curriculum of some other training project. When infon ation obtained from the 
case studies clarified these relationships, adjustments were made to the numbers obtained from the fiscal/1 ques- 
tionnaires. 
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Of the 28 SDAs in which information was 
available on the mix of services utilized by 
adults, 7 SDAs (25%) placed an extremely 
high emphasis on the provision of occupa- 
tional sldlls training, and between 91 and 
96% of their Title II-A-funded activities were 
either OJT or classroom training. These 
seven SDAs offered little or no direct job- 
search services to JTPA Title II-A par- 
ticipants as a "stand-alone" service (that is, 
job-search assistance only). In addition, of 
the seven, two SDAs offered no basic skills 
remediation services to Title II-A enroUees; 
two SDAs used other funding streams to 
provide basic skills remediation to Title II-A 
enroUees; and three SDAs integrated basic 
skills training into the curriculum of occupa- 
tional classroom training programs. 

At the other extreme were five SDAs 
(18%) that placed a low emphasis on occupa- 
tional skills training. In these SDAs, over 
50% of the adult activities in PY 86 were in 
basic skills remediation or direct job-search 
assistance. One might expect these two alter- 
native options to be in tension with each 
other; that is, that SDAs emphasizing direct 
job-search assistance would offer little in the 
way of basic skills remediation or vice versa. 
In fact, four out of these five SDAs em- 
phasized both direct job-search services and 
free-standing basic skills remediation 
projects. (This was also true of several other 
SDAs with more emphasis on occupational 
skills training.) 

The remaining 16 case study SDAs 
(57%) had more mixed service strategies. 
Eight SDAs had a dual emphasis on occupa- 
tional skills training and direct job-search as- 
sistance . The SDAs in this group did not 
ofler basic skills training as a distinct stand- 
alone service option funded by Title II-A, al- 
though some d'.d make basic skills training 
available by integrating it into occupational 
skills training curriculum or through another 
funding source. Five SDAs had a dual em- 
phasis on occupational skills training and 



basic skills training; with little emphasis on 
direct job-search assistance. Three SDAs 
could be characterized as using both basic 
skills training and direct job-search assis- 
tance to balance their primary focus on oc- 
cupational skills training. 

The Decision to Emphasize Occupation- 
al Skills Training. The SDAs that em- 
phasized occupational skills training to the 
exclusion of basic skills training or direct job- 
search assistance generally thought of occupa- 
tional skills uaining as the core service of the 
JTPA program. A number of these SDAs in- 
corporated basic skills remediation into 
specific occupational skills training projects 
but did not offer "stand-alone" remediation 
components. Similarly, these SDAs some- 
times offered training in job-search skills as 
part of occupational skills training cur- 
riculum but not as a free-standing service 
component. 

The SDAs' reasons for their emphasis on 
occupational skills training were diverse. 
Some SDAs lilced the convenience of "one- 
stop shopping," both for the clients and for 
the SDA. For clients, the existence of an in- 
tegrated service component addressing all of 
a given client's service needs reduced the 
likelihood that the client would get lost in the 
system and increased the likelihood that the 
client would progress smoothly from basic 
skills remediation to skills training to place- 
ment. For the SDA, the existence of a single 
service component for each client meant that 
costs per client were simpler to plan in ad- 
vance and that service-provider responsibility 
for placing the client at the conclusion of 
training was cleaier. 

For other SDAs, then* emphasis on oc- 
cupational skills training was based on an as- 
sessment that other activities were alieady 
available from other community resources 
and were more appropriately offered outside 
of Title II-A funding. A number of the case 
study SDAs were wary of being diverted 
from occupational skills training to address 
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the substantial deficits in basic skills ex- 
hibited by the JTPA applicant pool. Some 
said, '*Basic skills are the responsibility of 
the educational system, not of JTPA." 
Others carefully rationed the extent of basic 
skills remediation available through Title II- 
A funding and looked to other funding sour- 
ces, such as JTPA 8%, to respond to more 
extensive basic skills training needs. 

Although some SD As considered direct 
job-search services to be an appropriate 
response to the needs of some portion of 
their applicant pool, other SDAs made a 
policy decision to avoid "direct placement 
only'' services. The reasons for this position 
included the desire to differentiate JTPA 
from the placement-oriented function of the 
State employment service; the sense that job- 
search services were not intensive enough an 
intervention for most of the individuals in the 
JTPA applicant pool; and the position that in- 
dividuals who could be placed with direct 
job-search services could probably find jobs 
on their own without JTPA assistance. 

The Decision to Emphasize Remedia- 
tion in Basic Skills. Although only one of 
the case study SDAs offered free-standing 
basic skills training to more than 25% of 
their adult clients, a number of SDAs were in- 
creasing their emphasis on basic skills train- 
ing. The reasons for this were twofold. First, 
those SDAs that had a local goal of serving 
welfare recipients or other hard-to-serve 
groups saw basic skills remediation as an es- 
sential service component. Second, those 
SDAs in labor markets with declining un- 
employment rates found that ,eir more dis- 
advantaged applicant pool h. i a greater need 
for basic skills remediation. 

Because basic skills remediation was 
usually seen as only the first step in develop- 
ing client employabillty, to be followed by 
occupational skills training, the SDAs we 
visited were wary about how much basic 
skills training they could afford to offer. 
Their concerns had to do with both the 



higher cost of the sequence of services and 
the possibility that their entered-employment 
rate might be affected by clients who 
dropped out of the service program after the 
remediation phase. 

Usually SDAs decided to ration adult 
remediation through one of two methods. A 
number of SDAs established basic skill 
levels (reading and math performance) below 
which they decided to refer individuals to 
other eaucational resources rather than enroll- 
ing the individual in JTPA programs. This 
minimum entry-level requirement was set at 
different points in different SDAs, ranging 
from a fifth grade reading and math level to 
an eighth or ninth grade level. Only one of 
the case study SDAs specifically stated that 
it used Tide II-A funds to offer basic skills 
remediation to individuals reading below the 
fifth grade level. The second way of ration- 
ing the delivery of basic skills training was to 
set an absolute time limit on the duration of 
basic skills training that could be funded 
using Title II-A funds. Among the SDAs that 
used this method, the time limit ranged from 
6 weeks to 1 year. 

The Decision to Emphasize Job-Search 
Assistance. Eight of the case study SDAs 
(29%) placed an especially high emphasis in 
their service mix on the provision of job- 
search assistance as a stand-alone service. In 
these eight SDAs, job-search assistance (in- 
cluding direct placements) accounted for be- 
tween 35 and 62% of all activities received 
by adults. Given the limited intensity of most 
job-search assistance services, this pattem is 
somewhat troubling. Thus, we made an effort 
to trace why direct job-search services 
received such an emphasis in these SDAs 
and to what extent this design decision was 
influenced by the existence of the perfor- 
mance standards. 

For SDAs that are oriented toward per- 
forming well on the perfonnance standards 
without countervailing concems about serv- 
ing hard-to-serve clients or offering quality 
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services, an emphasis on job-search services 
could offer the opportunity to pursue "risk 
avoidance" objectives. First, if individuals 
are enrolled in job-search programs only 
after they obtain placements (as vv^as describ- 
ed in one of the case study SDAs), expanding 
this type of service could increase the overall 
entered-employment rate for the SDA. 
Second, if an SDA has difficulty or fears that 
it will have difficulty meeting the cost-per- 
entered-employment standard, emphasizing 
job-search assistance offers an inexpensive 
way to increase the number of entered 
employments. (However, SDAs using this 
strategy have discovered that it also tends to 
lower the average wage at placement.) 

However, despite our hypothesis that job- 
search assistance would be used at the ex- 
pense of services to hard-to-serve clients, 
only two of the eight SDAs with the greatest 
emphasis on job-search assistance used th;s 
design choice as part of a "risk avoidance*' 
strategy without offering more intensive ser- 
vices to clients with the greatest barriers to 
employment. One of these SDAs said that it 
screened JTPA applicants and referred those 
with the highest probability of success to 
classroom training in occupational skills and 
those with the least skills to a minimal job- 
search component, from which they were en- 
rolled in JTPA programs only if and when 
they located a job through their own efforts. 
The other SDA in this category was so 
oriented toward overachieving on the cost- 
per-entered-employment standard that it of- 
fered services of limited duration to all 
clients and did not utilize the savings from 
the job-search component to offer more cost- 
ly services to clients with greater skill 
deficiencies. 

The remaining SDAs that emphasized 
job-search assistance used it as part of a com- 
plicated "risk management" strategy. The in- 
clusion of job-search assistance services was 
direcdy or indirectly related to the SDAs' 
commitment to offer appropriate services to 



hard-to-serve clients. In two of these SDAs, 
special projects operated by community- 
based organizations that were committed to 
reaching hard-to-serve clients used direct job 
development to meet the needs of these 
clients. In four of the six SDAs, the unit cost 
savings associated with the job-search assis- 
tance component for the most job-ready 
clients provided enough leeway to permit the 
SDA to offer more basic skills remediation or 
other more intensive services to clients with 
greater employment barriers. Those case 
study SDAs that were most interested in ex- 
panding basic skills remediation to meet the 
needs of their applicant pools were among 
those SDAs that balanced the expense of 
these services with less expensive job-search 
assistance services. 

It is not clear what role job-search assis- 
tance would have played in the service mix 
offered by the case study SDAs in the ab- 
sence of the performance standards. Some 
SDAs would have discarded or reduced the 
size of this component. Other SDAs said ^hat 
job-search services had a definite place 
within the package of available services, 
even without the performance standards. 

In assessing the influence of the perfor- 
mance standards on the decision to em- 
phasize the delivery of job-search services, it 
is important to note that the eight SDAs with 
the highest emphasis on such services were 
performing substantially below their cost-per- 
entered-employment standard in PY 86. Six 
of these SDAs had achieved average costs of 
less than 60% of their cost standard, and all 
eight were below 76% of their cost standards. 
Thus, in theory, at least, they would have 
been able to substantially increase the inten- 
sity of their services even without the heavy 
emphasis on d.rect placements. 

Another possible hypothesis is that State 
policies encouraging overperformance on the 
standards rather than the standards themsel- 
ves were responsible for the high emphasis 
on job-search assistance in some SDAs. 
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However, only two of the eight SDAs with 
the greatest emphasis on direct placements 
are from States that reward overperformance 
on the standards, which is consistent with the 
quantitative results that found policies em- 
phasizing exceeding standards did not sig- 
nificantly increase the amount of job-search 
assistance. 

Thus, if the standards are responsible for 
the use of job-search assistance services, it is 
not through any direct effect of cost limita- 
tions imposed by the performance standards. 
It is still possible that SDAs are responding 
to a misperception that the standards require 
low-cost services or through a sense of com- 
petition among SDAs for high measured per- 
formance that feeds on itself. 

As discussed elsewhere in this report, we 
found that the reduction in overall JTPA 
funding levels associated with declining un- 
employment rates in 24 of the 30 (80%) case 
study sites was at least as important as the 
performance standards in causing SDAs to 
reduce unit costs. In the environment of 
declining funds, many of the case study 
SDAs felt compelled to reduce unit costs to 
stretch the reduced funds over as many par- 
ticipants as possible. 

Three of the eight SDAs with the greatest 
emphasis on job-search services in their ser- 
vice mix are planning to shift toward a more 
intensive service mix in PY 88. One SDA 
has identified the introduction of follow-up 
standards as the impetus to make the shift. 
The other two SDAs say they "overshot" 
their goal in trying to design co.st-effective 
services and would like ,0 make the ser/xces 
more intensive in response to client needs. 

Variaiions in the Use of OJT Versus 
Classroom Training 

Another important dimension of the 
design of SDA service packages is the rela- 
tive emphasis on classroom training Vftrsus 
OJT within the occupational skills trainihg 
category. 

The extent to which SDAs emphasized 



OJT versus classroom training varied 
dramatically across the 30 case study sites. 
Of the 29 SDAs for which percentages could 
be computed, 10 SDAs (34%) strongly em- 
phasized classroom training as a mode of oc- 
cupational skills training, with classroom 
training accounting for 75 to 100% of all oc- 
cupational training. Reasons for this em- 
phasis included the following: 

• OJT was not appropriate for the 
characteristics of the applicant pool, 
which had shifted to less job-ready 
clients needing more remediation or 
more intensive services (4 SDAs). 

• OJT was not viewed as being ap- 
propriate for the "full employment" 
economy in which employers were 
desperate to find workers (1 SDA). 

• Classroom training was viewed to be 
a "better buy" because it enabled 
SDAs to leverage non-JTPA funds 
through purchasing services fi-om 
publicly funded educational institu- 
tions (1 SDA). 

• Classroom training v/as viewed as 
more appropriate to the needs of ^'"'^ 
desired target group — welfare 
mothers (1 SDA). 

• OJT was used only for a very specific 
purpose, to peisuade employers in 
new downtown developments to hire 
JTPA clients (1 SDA). 

• The SDA had not shifted from the ser- 
vice mix used under CETA which em- 
phasized classroom training (2 SDAs). 

At the other extreme, another 10 SDAs 
(34%) utilized a particularly high proportion 
of OJT within their service packages, v/iih 
OJT accounting for over 50% of the occupa- 
tional skills training total. The most common 
explanation given for this emphasis was '''be- 
cause tlie PIC likes OJT." Tliis explanatior* 
was offered by five SDAs. Other reasons for 
emphasizing OJT in the ser/ice mix were 
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• Employers were really interested in 
having the SDA recruit and screen 
workers in a tight labor market 
(2SDAS). 

• Clients who needed immediate in- 
come preferred OJT (1 SDA). 

• OJT was relatively inexpensive com- 
pared with classroom training 

(3 SDAs). 

• The number of potential classroom 
training providers was very limited 
(1 SDA). 

• OJT was vievv^ed as appropriate for 
individuals with limited work ex- 
perience who wouldn*t be able to suc- 
ceed in classroom training (2 SDAs). 

• OJT was attractive because a high 
placement level is practically guaran- 
teed (2 SDAs). 

• OJT was part of a strategy to secure 
higher wage jobs for those with low 
skills (2 SDAs). 

The remaining nine SDAs (31%) utilized 
a mix of occupational skills training in which 
OJT accounted for between 26 and 50% of 
the total of classroom training plus OJT ac- 
tivities. In these SDAs, not surprisingly, less 
clear-cut reasons were given for preferring 
one mode of occupational training over 
another. 

None of the reasons given by the case 
study SDAs about why they decided to em- 
phasize or de-emphasize OJT referenced the 
performance standards directly. This design 
decision is based more on other factors — PIC 
input, the historical use of specific service 
providers, and the extent to which OJT is felt 
to be appropriate to the needs of the applicant 
pool — than on the existence of performance, 
standards. Again, the quantitative results are 
consistent: State policies emphasizing exceed- 
ing standards did not significantly affect the 
amount of OJT services provided. 

In some SDAs, OJT was a cheaper ser- 



vice component to operate than classroom 
training. Thus, the desire to spread limited 
funds over more enroUees also came into 
play in the decision about how much to em- 
phasize OJT. SDAs varied dramatically in 
how they view OJT, including whether it was 
seen as a service to employers or clients, 
whether it was viewed as increasing employ- 
ment skills, and whether it was viewed as ap- 
propriate for more job-ready or less 
job-ready clients. These variations were ac- 
companied by variations in the average dura- 
tion of OJT (from 3 weeks in some case 
study SDAs to an average of 6 to 9 months in 
other SDAs). 

Variations in the Intensity and 
Duration of JTPA Services 

One of the most important dimensions of 
service design examined for this study was 
the variation in the intensity of the services 
offered to individual participants. One of the 
accusations leveled at the performance-stand- 
ards system by its critics is that the perfor- 
mance standards have caused SDAs to 
emphasize short and superficial service inter- 
ventions. The case studies offered informa- 
tion about the average duration of various 
service components and how SDAs decided 
how much of each service would be available 
to JTPA enrollees. In addition, the case study 
materials described variations across SDAs 
in the extent to which individual enrollees 
were sequenced through more than one ser- 
vice or enrolled in only a single service. 
Finally, the case studies explored the extent 
to which additional resources were being 
used to supplement the services paid for with 
Title II-A funds. Taken together, these dimen- 
sions of service design comprise the richness 
(or poverty) of the JTPA program in respond- 
ing to the needs of individuals with severe 
employment barriers. 

There was tremendous variation in the 
average duration of individual services of- 
fered by the case study SDAs. In some 
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SDAs, OJT averaged 3 v/eeks in length; in 
other SDAs 8 months was the average dura- 
tion and 12 months was the maximunr dura- 
tion of OJT. Similarly, classroom training 
ranged from an average length of 13 weeks 
in some SDAs to an average length of 9 
months to 1 year in others. Several SDAs 
would pay for up to 2 years of occupational 
classroom training, and one SDA even 
authorized 4-year classroom training plans. 

Average figures on total time enrolled in 
a program were not very helpful in disentan- 
gling the variations across SDAs because 
these averages depended, in turn, on how 
much variation there was within an individ- 
r.al SDA in the length of time individuals are 
enrolled. In SDAs that utilized direct place- 
ment components, the short duration of these 
services (usually less than 1 month and as 
short as 1 day) made the average duration 
across the SDA short even if occupational 
skills training components offered relatively 
intensive services to other participants. 

The intensity of the service package of- 
fered in each SDA was influenced by a series 
of factors, including 

• The availability of funds to supple- 
ment Title II-A funds in purchasing 
services for enroUees, 

• The costs of the service delivery op- 
tions used in that SDA (including the 
cost of operation of various service 
providers and the costs of providing 
various service modules), 

• The SDA's goals about what types of 
clients it wanted to serve and the in- 
tensity and types of services they re- 
quired, 

• The desire to offer JTPA services to 
as many participants as possible, 

• The cost constraints created by the 
cost-per-entered-employment stand- 
ard (and to a lesser extent the cost- 
per-positive-termination standard), 



and 

• The extent to which the SDA wants 
to meet or exceed its cost standard. 
We found the following typology useful 
in summarizing the variations across SDAs 
in service intensity. The average cost to Title 
II-A of providing services enters into this 
typology because it represents the resource 
constraint within which SDAs are making 
design decisions. SDAs appear to fall into 
the following five categories in terms of the 
intensity of the services they offer: 

Type 1 : short and inexpensive services; 
Type 2: long and inexpensive services; 
Type 3: much variation across ser- 
vices (some short and inexpen- 
sive; some long and 
expensive); 
Type 4: little variation across services; 
and 

Type 5: short and expensive services. 

Type 1 SDAs — ^with short and inexpen- 
sive services — of which we identified two 
(7%) within our case study sample, spent be- 
tween $2,200 and 2,300 per adult-entered- 
employment outcome. For both SDAs, this 
expenditure level was substantially lower 
than their cost standard, reaching only 60% 
or less of the average cost that would have 
been permitted by the cost standard. 

For this average expenditure, one SDA 
offered a 3- to 4-week OJT program, a high 
proportion of job-search assistance, a half 
day pre-employment YEC curriculum, up to 
10 weeks of remedial education, and occupa- 
tional classroom training to only 15% of all 
adult terminees. The other SDA in this 
category offered slightly longer services: 
OJT lasting an average of 8 weeks, class- 
room training lasting up to 16 weeks, and be- 
tween 30 and 40% of all terminees receiving 
job-search assistance only. Although one 
SDA did not emphasize supplementing JTPA 
Tide II-A expenditures with other funding 
sources, the other Tj-pe 1 SDA gave priority 
to service providers that could use alternative 
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funding sources to support their particular 
projects, resulting in low unit prices for 
delivery of quality services. 

Type 1 SDAs, with relatively inexpensive 
and short-term services, appear to be highly 
influenced by a desire to exceed their cost 
standards in order to maximize their incen- 
tive award. Both Type 1 SDAs are located in 
States where "overperforming" on the stand- 
ards results in the highest possible incentive 
awards, and staff from both SDAs reported 
that they were motivated by a desire to get 
the largest incentive award possible. Thus, 
the outcome in these SDAs reflects a rather 
one-sided emphasis on low costs, without a 
balancing concern for obtaining quality jobs 
or reaching hard-to-serve clients. 

Both Type 1 SDAs decided after the fact 
that their program design during PY 86 had 
been an overreaction to the cost standards. 
They both were considering shifting their 
program mix in subsequent years to increase 
the intensity of the services provided. One 
SDA was planning to cut back on the propor- 
tion of job-search services; the other was 
trying to decide whether to increase the 
amount of basic skills remediation or to in- 
crease the intensity of occupational skills 
training. 

Type 2 SDAs — ^with long aiiJ inexpen- 
sive services — were more frequent than Type 
1 SDAs. Six case study SDAs (20%) belong 
in this category. All of these SDAs had ex- 
ceptionally low costs per entered employ- 
ment (ranging from 33 to 56% of their 
standards) but were able to offer particularly 
long or enriched program alternatives to their 
enroUees. Five of these six SDAs used public 
educational institutions (primarily com- 
munity colleges) to offer classroom training 
to JTPA enroUees. Three of these six SDAs 
were from a State where community college 
tuition cost less than $100 per quarter and 
were thus able to offer at least 26 weeks and 
often up to 2 years, or in one SDA, even 4 
years of classroom training to JTPA par- 



ticipants. 

The OJT duration ranged from 6 months 
to 1 year in these SDAs. (It is not clear in this 
case whether the intensity of services 
benefited the employers, the trainees, or 
both.) One of the six Type 2 SDAs was able 
to offer an enriched package of services be- 
cause it supplemented JTPA resources with 
funding from a State welfare initiative and, in 
addition, obtained low tuition from a public 
educational institution for classroom training. 

Four of the six Type 2 SDAs emphasized 
"sequencing" from one service to another for 
individual enroUees. The sequencing was 
primarily from basic skills remediation to oc- 
cupational skills training, but in several 
SDAs it involvexi more complicated sequen- 
ces, including, for example, basic skills train- 
ing, followed by work experience, foUowed 
by OJT. In one of the Type 2 SDAs, about 
30% of the individuals who received class- 
room training went on to participate in OJT 
before being placed. 

Type 2 SDAs, which offer long-term and 
inexpensive services, were fortunate that the 
existence of the cost standards did not con- 
strain their ability to offer intensive services. 
Many of these SDAs were able to take ad- 
vantage of inexpensive classroom skills train- 
ing from public educational institutions or 
were able to supplement Title II-A funds 
with funds from a State work-welfare initia- 
tive. A number of these SDAs said that the 
performance standards had no effect on the 
development of their program design. Al- 
though they were able to offer long-term 
classroom training, these SDAs were not par- 
ticularly notable in their efforts to recruit pai- 
ticularly hard-to-serve client groups who 
might have found it difficult to take ad- 
vantage of training from mainstream educa- 
tional institutions. 

Type 3 SDAs — with much variation 
across services — included several of the 
SDAs that utilized large proportions of job- 
search assistance but that took advantage of 
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the cost savings generated by this component 
to offer richer occupational skills training ser- 
vices. (Although we did not find SDAs that 
offered long and expensive services to all par- 
ticipants, SDAs in this category offered such 
services to selected clients.) We identified 
five Type 3 SDAs (17%). Like Type 1 and 
Type 2 SDAs, these SDAs were performing 
well on their cost-per-entered-employment 
standards (with costs ranging from 30 to 
62% below these standards in PY 86). On 
average, these SDAs offered up to 6 months 
of classroom training. Three of the five 
SDAs also offered substantial basic skills 
components within Title II-A funded ac- 
tivities, and a fourth had an extensive basic 
skills curriculum funded from other sources. 
In two of the five SDAs, welfare system 
funding was used to supplement JTPA fund- 
ing in enriching the package of services avail- 
able to qualified JTPA participants. 

Three of the five Type 3 SDAs, which 
combined an emphasis on job-search assis- 
tance for some clients with the ability to 
offer more intensive services to other clients, 
were central city SDAs with an identifiable 
pool of hard-to-serve clients who needed 
basic skills remediation and other more inten- 
sive services to obtain employment. One of 
the five SDAs in this category, however, had 
very little commitment to reaching hard-to- 
serve clients and selected the most employ- 
able clients into the intensive services, 
leaving the most needy clients to find jobs by 
themselves using the job-search component. 
All five SDAs in this category escaped direct 
pressures from the cost-per-entered-employ- 
ment standard by using job-search services to 
lov/er unit costs. Perhaps as a result, two of 
the five had a hard time meeting their 
average wage-at-placement standard. 

Type 4 SDAs — with little variation 
across services — tended to stretch the avail- 
able funds as far as they couid but were start- 
ing to reduce the duration of different 
services to cut costs. We included 15 SDAs 



(50%) in this category. Because these SDAs 
did not emphasize job-search services to 
generate some low-cost placements, they en- 
sured that they met their cost objectives by 
limiting the duration of the rest of the ser- 
vices they offered. The average duration of 
classroom training in these SDAs ranged 
from 4 to 6 months, and OJT averaged 3 
months. Basic skills remediation was often 
felt to be an important service in these areas, 
but time limits were usually placed on the 
receipt of remedial training to ensure that 
cost limits were not exceeded. 

Thirteen of the 15 SDAs in this category 
used some other funding source to supple- 
ment the expenditures from Title II-A funds. 
Welfare system funds were used to supple- 
ment Title II-A funds in four SDAs; JTPA 
8% funds were used to support additional ser- 
vices to Title n- A enroUees in two SDAs; 
vocational rehabilitation funds were used to 
supplement Title II-A funds in serving hand- 
icapped participants in two SDAs; and public 
educational institutions offered a cost ad- 
vantage as classroom training providers in 
eight SDAs. Despite these additional resour- 
ces, the Type 4 SDAs were spending at 
levels closer to their cost-per-entered-employ- 
ment standards than other SDAs. Only 2 of 
the 15 SDAs were under 60% of the cost 
standard. Five Type 4 SDAs were between 
60 and 80% of their cost standard. Six SDAs 
were between 80 and 100% of the cost stand- 
ard, and two SDAs in this category had ex- 
ceeded their cost-per-entered-employment 
standard, one by 40% and one by 28%. 

Type 5 SDAs — with short and expensive 
services— are similar to Type 4 SDAs except 
they have reached their funding limits at rela- 
tively meager levels of service. We identified 
two Type 5 SDAs (7%). One SDA limited 
classroom training duration to 13 weeks; the 
other averaged 12 to 16 weeks for both class- 
room training and OJT. What is distinctive 
about these SDAs is that their average cost 
per entered employment for adults in both 
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cases exceeds $4,000 for this low level of ser- 
vice. Their costs are 89 and 74% of their cost 
standards, respectively, without much room 
to add more intensive services within the 
standards limits. One of these SDAs used 
public educational institutions that give JTPA 
participants a reduced tuition rate; the other 
did not supplement JTPA Title II-A resour- 
ces to enrich its service package or to reduce 
its costs. 

The 15 Type 4 SDAs and the 2 Type 5 
SDAs were beginning to feel pinched (and in 
some cases squeezed) by the cost-per-entered- 
employment standard. In these cases, the per- 
formance standards were directly responsible 
for reductions in the duration of specific train- 
ing components and the deletion of training 
in some occupations that were viewed as 
more expensive than the SDA could v*;ford. 
Although they wanted to respond to the 
needs of hard-to-serve clients, these SDAs 
were struggling with how to make tradeoffs 
between offering more remedial education 
(but not more than they could afford) and of- 
fering more intensive occupational training. 
SDAs in rural areas or without the ability to 
leverage other funding sources were par- 
ticularly pressured because of the cost limits 
represented by the standards. Nevertheless, 
only 2 of these 17 SDAs failed to meet the 
cost standard in FY 86, and neither was par- 
i jularly upset about missing the standard. 

Variations in the Extent and Types 
of Specialized Youth Programs 

The 30 case study SDAs varied widely in 
the percentage of youth served, ranging from 
21 to 60%. The case study SDAs also varied 
widely in the extent to which youth were 
served in mainstream (adult) occupational 
skills training programs. In most cases, when 
youth did participate in occupational skills 
training programs, they did so alongside 
adult JTPA participants (although one SDA 
established a separate youth OJT contract to 
make sure that youth had access to this ser- 



vice option). Of the 25 SDAs for which data 
are available, 8 SDAs (32%) have a high em- 
phasis on occupational skills training for 
youth , with c /er 60% of all youth activities 
consisting of either classroom training or 
OJT. Another 13 SDAs (52%) have a low em- 
phasis on occupational skills training for 
youth , with less than 30% of all youth ac- 
tivities consisting of either classroom training 
or OJT. The remaining 4 SDAs (16%) have a 
more mixed service strategy for youth with 
from 30 to 60% of all youth activities consist- 
ing of either classroom training or OJT. 

When they did not participate in main- 
stream occupational skills training, youth par- 
ticipated in a variety of activities, including 
remedial basic skills training (either youth 
specific or mainstream), job-search assis- 
tance (either youth specific or mainstream), a 
variety of projects oriented toward achieving 
youth employment competencies, and work 
experience or tiyout employment activities. 

Factors that influenced SDAs in their 
design of youth programs were 

• The Federal requirement for 40% 
youth expenditures and the role 
played by the States in adjusting and 
monitoring this requirement; 

• The extent to which there was local 
PIC support for developing programs 
for in-school youth; 

• The extent to which the local school 
districts were willing to coordinate 
with JTPA programs to identify 
prospective participants and even to 
integrate JTPA-funded programs into 
the regular school curriculum; and 

• Whether SDAs had trouble meeting 
its youth performance standards, and 
if so, whether the SDA desired to 
make changes in its program design 
or implementation practices to im- 
prove its measured performance on 
the youth standards. 

In some SDAs, specialized youth 
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programs were designed to achieve YECs as 
the desired objective. In other SDAs, YECs 
were tracked, but the primary SDA objective 
was to place youth in employment at the con- 
clusion of training. Most of the SDAs we 
visited targeted YEC-oriented programs 
toward in-school youth and used employ- 
ment-oriented programs for out-of-school 
youth. However, some of the case study 
SDAs integrated YEC reponing systems into 
all their occupational skills training programs 
so that youth served by any of these 
programs could get credit for participants' 
achieving youth competencies as well as ob- 
taining employment. At least 4 of the 30 case 
study SDAs (13%) expressed dislike (one 
labeled it "disdain") for the YEC system 
promulgated by DOL, and seven SDAs 
(23%) said they preferred to orient all their 
youth programs toward achieving exixpioy- 
ment. However, six of these seven SDAs L.,u 
implemented YEC systems to capture the 
necessary youth positive-termination out- 
cc5mes to meet the positive-termination-rate 
standard. 

As with the design of the adult activities, 
SDAs were influenced initially and primarily 
by their local goals on which groups of youth 
they wanted to serve and whether they 
wanted to operate a separate set of youth- 
oriented programs. Federal youth expendi- 
ture requirements, as adjusted and monitored 
by the States, were an important influence on 
program design for a number of SDAs. At 
least four of the case study SDAs (13%) 
made a point of saying that without the youth 
expenditure requirement, they would have 
devoted fewer resources to youth programs. 
Particularly in tight labor markets, youth 
were seen as difficult to attract into training 
programs and difficult to retain in those 
programs until the completion of training. 

Furthermore, SDAs that were trying to 
target "high-risk" youth found this group 
more unpredictable and less likely to achieve 
good outcomes than comparable "high-risk" 



adults. In other SDAs, the youth expenditure 
requirement was consistent with a local 
desire to place a high priority on services to 
youth and did not alter the locally devised 
program design. 

Equitable service requirements for 
dropouts also influenced youth programs 
and, when emphasized by State monitors, 
caused some SDAs to try to target youth 
dropouts by using special projects. These ef- 
forts were not always successful. One case 
study SDA told of funding a special project 
for youth dropouts that succeeding in enroll- 
ing only six youth and placing only one par- 
ticipant into a job. (This project was 
cancelled after about 6 months.) 

Of the 30 case study SDAs, 27 (90%) 
reported using some specialized youth 
programs under Title 11- A. In the summary 
that follows, some SDAs' approaches fit 
under more than one category. 

• Fourteen SDAs (52%) were operating 
programs for in-school youth. Of 
these 14 programs, 7 were oriented 
toward achieving YECs, 4 were 
oriented toward employment out- 
comes, and the orientation of 3 was 
not known. 

• Ten SDAs (37%) used work ex- 
perience or tryout employment as 
part of their specialized youth 
programs, followed by placement in 
part-time employment (for in-school 
youth) or full -time employment (for 
out-of-school youth or youth about to 
graduate). 

• Five SDAs ( 19%) oriented at least 
part of their special youth programs 
to dropout prevention projects. One 
SDA designed an intensive 4-ye9r 
program for potential dropouts, start- 
ing from age 14 through graduation. 
One SDA used direct placement of 
youth in part-time employment as 
part of a dropout reduction strategy 
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for in-school youth. 

• Twelve SDAs (44%) offered pre- 
employment/work maturity YEC cur- 
ricula for out-of-school youth. Of 
these projects, three were particularly 
short term (one lasted a h^ day, one 
lasted 4 days, and one lasted 3 
weeks). Two of the three were in- 
tended to be used as a front-end ser- 
vice followed by a more intensive 
intervention. The remaining nine 
YEC curricula were longer term 
projects. 

• Five SDAs (19%) offered other spe- 
cial projects for out-of-school youth. 
Of these five projects, four were 
oriented to remediation in basic skills 
and obtaining a GED, and one was a 
youth-specific OJT project. 

The existence of performance standards 
for youth did not seem to affect the extent of 
SDA resources devoted to youth programs 
per se. However, the performance standards 
for youth were more problematic for many 
SDAs at the outset of JTPA than the adult 
standards, and more "fine-tuning" program- 
design responses resulted than with the adult 
standards. This was due to the differences be- 
tween programs offered to in-school versus 
out-of-school youth. Staff and PICs in many 
SDAs were engaged in intensive debates 
about the extent to which J ^PA programs 
should be offering employment-oriented or 
competency-oriented training. SDAs that 
chose to emphasize competency-oriented ser- 
vices often found it difficult to meet their 
entered-employment-rate standard, while 
those emphasizing employment-oriented 
programs often found it difficult to meet their 
positive-termination-rate standard. 

In some cases, this tension between the 
standards influenced the balance between 
employment-oriented and competency- 
oriented projects: 



• Three of the case study SDAs said 
they had added a YEC system only 
after missing the positive-termination- 
rate standard the first year of JTPA. 

• Two SDAs had implemented "quick 
and dirty" front-end YEC curricula 
that they required of all youth enter- 
ing JTPA programs, to ensure that the 
SDAs could meet their youth positive- 
termination-rate standard. 

• One SDA was still resisting im- 
plementing a YEC system, despite 
missing tWO youth standards several 
years in a row after getting a State ad- 
justment because it was operating an 
employment-oriented program for 
youth. 

• One SDA decided to exclude 14- and 
15-year-old youth from its in-school 
youth program after discovering that 
serving this age group substantially 
lowered tl?eir cost-per-positive-ter- 
mination standard. (They felt they 
were serving a very troubled youth 
population and one that required 
more intensive services than usual, 
rather than less intensive as the model 
indicated.) 

• One SDA used 6% money (exempt 
from the standards in that State) to 
operate a program for high-risk youth 
because it felt the positive-termina- 
tion-rate standard was set too high to 
serve high-risk youth under the perfor- 
mance standards. 

In our case study SDAs in PY 86, two 
SDAs missed the youth entered-employment- 
rate standard (one by only a tiny amount), 
one SDA missed the positive-termination- 
rate standard, and two SDAs missed the cost- 
per-positive-termination standard. Two 
SDAs missed two youth standards, and the 
rest missed only one. In response to these per- 
formance difficulties: 
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• One SDA planned to totally redesign 
its in-school youth program and con- 
tracting practices with youth 
providers to obtain better control over 
the outcomes generated by the 
program. (This was one of the two 
SDAs that missed two youth stand- 
ards and is also the SDA that has 
resisted implementing YECs.) 

• One SDA had missed four standards, 
including all the youth standards, and 
was planning to increase the number 
of different service providers and cut 
the size of risky programs in response 
to its performance. 

• One SDA was not really interested in 
a large incentive award and did well 
on the other standards, so it was not 
too perturbed by missing the youth 
entered-employment-rate standard, al- 
though it only achieved 23% of 
entered employments for youth in- 
stead of the required 32%. 

• One SDA that missed its cost-per- 
positive-termination standard decided 
to stop serving 14- and 15-year-old 
youth in its in-school program. 

At the time of our site visits, the case 
study SDAs could not yet predict how the 
new configuration of youth standards for 
PY 88 would alter their incentives or their 
youth program-design strategies. It is interest- 
ing to note, however, that relatively few of 
the case study sites had implemented YECs 
for basic skills training or occupational skills 
training, and the few that had more than one 
type of YEC rarely measured deficiencies or 
attempted to correct deficiencies in more 
than one area for any given JTPA participant. 

Variations in the Costs 
of JTPA Services 

As the quantitative analysis has shown, 
the costs of providing JTPA services varied 
dramatically from SDA to SDA. Within the 



case study sample, the average cost per 
entered employment for adults in PY 86 
ranged from $1,334 to 7,200, and the average 
cost per youth positive termination ranged 
from a low of $1,250 to 5,699. 

The SDAs that we visited did not con- 
sider achieving a specific average cost as 
valuable in and of itself. Rather, SDAs set 
cost goals that set forth the resource con- 
straints within which other local goals had to 
be addressed. The availability of additional 
resources to supplement JTPA expenditures 
and the cost of purchasing training from ex- 
isting educational institutions were important 
local factors shaping the average costs 
reported by the JTPA system. Furthermore, 
as discussed in the previous section, average 
cost figures often disguised tremendous varia- 
tion within a given SDA in the duration and 
cost of services received by different JTPA 
participants. 

Nevertheless, SDAs were very concerned 
about reducing average JTPA costs for two 
reasons. First, a majority of the SDAs in the 
case study sample (and in the Nation) were 
experiencing declining unemployment rates 
during PY 86. Because the JTPA Title II-A 
funding allocation formula is very sensitive 
to local unemployment rates, this meant that 
their total funding levels were getting smaller 
from year to year. For some SDAs, the level 
of JTPA funding had dropped by 50% since 
the beginning of JTPA, and allocations for 
PY 87 and PY 88 were continuing to drop 
sharply. 

Many of these SDAs were reaching a 
funding level that required a dramatic 
retrenching and program redesign. Others 
were able to get by with essentially the same 
program design by reducing the size of in- 
dividual projects and putting pressure on con- 
tractors to reduce costs. Whichever 
program-design response was necessary, the 
SDAs facing reduced training funds had to 
decide whether to serve a smaller volume of 
participants and keep the average cost of ser- 
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vices stable or to serve as many participants 
as before by cutting the average cost of the 
services received. Almost universally, the 
SDAs in the case study sample had decided 
to reach as many individuals as they possibly 
could without diluting the service intensity 
beyond reason. 

A total of 16 of the 30 SDAs visited 
(53%) were in the process of substantial 
retrenchment due to declining Title II-A fund- 
ing streams. Responses included the follow- 
ing: 

• Looking for opportunities to leverage 
JTPA funds with other funds, 

• Funding special projects that ap- 
peared to be particularly cost-effec- 
tive, 

• Limiting the duration of training 
programs (for example, cutting the 
length of specific training projects or 
reducing the average length of class- 
room training or OJT across the 
board), 

• Shifting the program mix from more 
expensive to less expensive service 
components, 

• Eliminating particular training 
programs that were viewed as being 
too expensive (for example, a 2-year 
electronic technician training program 
in one SDA and licensed vocational 
nurse training in several SDAs), 

• Putting pressure on contractors to 
reduce their unit costs, 

• Utilizing contractors that had a track 
record of fulfilling their goals, and 

• Trying new programs cautiously and 
deleting them if they did not seem to 
be working. 

The second reason that SDAs were con- 
cerned about lowering their average costs 
was the existence of the cost performance 
standards. However, most SDAs in the case 
study sample were already performing at 



levels that were far below (better than) their 
cost standards. Thirteen of the case-study 
SDAs (43%) had costs per entered employ- 
ment that were at or below 60% of their cost 
standard in FY 86. Only 5 (17%) SDAs had 
expenditures above 90% of tiieir cost stand- 
ard. (Two of these SDAs failed their adult 
cost standard in PY 86.) Altiiough SDAs 
may have adjusted their program mix initial- 
ly to make sure they would meet tiieir cost 
standards, the actual standards themselves 
were not at a level in PY 86 tiiat would have 
caused additional reductions in service inten- 
sity and cost. 

However, as discussed in the quantitative 
analysis, some States award incentive funds 
according to a formula tiiat (1) rewarded 
SDAs according to the extent tiiey exceeded 
a standard, witii a higher award going to 
SDAs that exceeded their standard by a 
greater proportion, and (2) unintentionally 
gave greater weight to the cost standards than 
other standards by averaging the extent all 
standards were exceeded (because there was 
more potential for exceeding tiie cost stand- 
ards by a higher percentage than there was 
for other standards). Some SDAs facing these 
incentives chose to exceed tiieir cost stand- 
ards (that is, spend less than the standards) 
by a great deal to maximize their incentive 
awards. Under these circumstances, the cost 
standards levels took on a heightened 
management significance for the SDA. 

Summary and Conclusions 

Performance standards had littie in- 
fluence on the service mix offered to adults 
by SDAs in tiie case studies. A few SDAs 
mentioned the standards when explaining 
their emphasis on OJT over classroom train- 
ing. However, the bigger influences on ser- 
vice mix were 

• The SDA's historical pattern of ser- 
vice delivery, as evolved from the 
CETA program, 

• The SDA's goals for serving different 
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client groups, and 

• PIC preferences about what services 
to emphasize in the SDA service 
package. 

SDAs exhibited wide variation in the in- 
tensity and duration of JTPA services; the 
case study sites fell into five categories: 

• Short and inexpensive services 
(2 SDAs, 7%), 

• Long and inexpensive services 
(6 SDAs, 20%), 

Much variation across services 
(5 SDAs, 17%), 

• Little variation across services 
(15 SDAs, 50%), and 

• Short and expensive services 
(2 SDAs, 7%). 

The existence of performance standards 
for youth did not seem to affect the extent of 
SDA resources devoted to youth programs 
perse. Local goals about whether to run 
employment-oriented or competency- 
oriented programs for youth as well as the 
Federal requirement to spend 40% of JTPA 
funds on youth had greater influences than 
the standards. 

Some SDAs felt that the existence of the 
two outcome standards for youth unduly con- 
strained their youth programming. SDAs that 
offered competency-oriented programs ouen 
found it difficult to meet their entered- 
employment-rate standard, and SDAs offer- 
ing employment-oriented programs often 
found it difficult to meet their positive-ter- 
mination-rate standard. 

Although a number of SDAs had insti- 
tuted YEC systems only after missing their 
positive-termination-rate standard, one SDA 
was still resisting implementing such a sys- 
tem in spite of continuing to miss the stand- 
ard. 

The cost standard had some influence on 
service design because SDAs often wanted to 
not only meet but exceed this standard to 



maximize their 6% awards. However, other 
factors were equally as imponant in determin- 
ing the intensity and duration of services. 
These included 

• The availability of funds to supple- 
ment Title II-A funds in purchasing 
services, 

• The cost of the various service pack- 
ages available from the service 
providers in the SDA, 

• The SDA's goals about the types of 
clients it wanted to serve and the in- 
tensity and types of services that they 
required, and 

• The desire to spread JTPA funds over 
as many clients as possible. 

The unit costs of providing JTPA ser- 
vices varied dramatically between SDAs, and 
average cost figures often disguised substan- 
tial variation within SDAs in the duration 
and intensity of services received by dif- 
ferent participants. SDAs were very con- 
cerned about reducing average JTPA costs 
for two reasons. 

• First, a majority of SDAs were ex- 
periencing declining JTPA alloca- 
tions (due to declining unemployment 
rates), which required a decision 
about whether to serve a smaller 
volume of participants and keep the 
average cost of services stable or to 
try to serve as many participants as 
before by cutting tht average cost of 
the services. Almost universally, the 
SDAs visited had decided to try to 
reach as many individuals as possible 
without diluting Ae service intensity 
beyond reason. 

• Second, some SDAs desired to ex- 
ceed the cost performance standard. 
Although the vast majority were in no 
danger of missing this standard, the 
emphasis on overperforming grew 
out of a desire to maximize their in- 
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centive awards. 

The qualitative analysis found that perfor- 
mance standards had unintended effects only 
in SDAs without well-defined client and ser- 
vice goals, which were rare. We found, for 
example, only two SDAs that chose to offer 
substantial amounts of job-search assistance 
as a risk avoidance response to performance 
standards. We conclude that only 10 to 20% 
of the SDAs in the case study sample made 
substantial modifications to their program 
design in response to performance standards. 
Although the 30 case study SDAs were 
chosen to be as representative of all SDAs as 
possible, we studied only a small sample and 
it is possible that performance standards had 
unintended effects in a different proportion 
of all SDAs. 

We classified SDA reactions in the case 
study sample using information from the in- 
depth interviews that is not available in the 



quantitative survey of all SDAs. Nonethe- 
less, the quantitative information available is 
consistent with the qualitative conclusions. 
For example, we find that the average effects 
of performance standards are quite modest, 
which is consistent with the qualitative find- 
ings of large impacts in only a small number 
of SDAs. We also found that the proportion 
of SDAs offering substantial amounts of job- 
search assistance is similar in the qualitative 
and quantitative samples. Furthermore, the 
director questionnaire asked directors to rate 
how influential performance standards were 
in their choice of services, and approximately 
20% responded that performance standards 
were extremely influential. Thus, although 
we cannot say with certainty that the propor- 
tion of case study SDAs with substantial reac- 
tions to performance standards can be 
generalized to all SDAs, the information 
available in the quantitative sample is consis- 
tent with that in the qualitative sample. 
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Introduction 

As demonstrated in the quantitative 
analysis, SDAs' choices about the procedures 
for enrolling clients into program services 
and choices about how to deliver services af- 
fect clients, services, and costs. Furthermore, 
some SDAs' choices about management prac- 
tice were influenced by performance stand- 
ards. In the case studies, we explored the 
various ways that SDAs chose to manage the 
JTPA program and the factors that influenced 
SDAs* choices of management practices. 

Overview of Chapter 
and Key Findings 

In this chapter, we begin by describing 
SDA management goals and some of the 
management practices that SDAs adopted in 
response to concerns about specific perfor- 
mance standards. Most of these practices rep- 
resent risk management rather than risk 



avoidance strategies. 

We next describe the major variations in 
SDA practices in several management areas, 
including 

• SDA and service-provider practices 
in assessing applicant skills, assigning 
clients to particular services, and en- 
rolling clients. Most case study SDAs 
maintained centralized control over 
client intake. The extent of SDA con- 
trol, however, was not related to the 
extent of service to the hard to serve. 

• Division of responsibilities between 
SDAs and service providers concem- 
ing placement, and practices affecting 
the timing of client termination. Most 
SDAs give service providers primary 
responsibility for placing clients. 

• SDA practices surrounding the selec- 
tion of service providers. SDA 
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decisions about retaining providers 
and the size of their funds was af- 
fected by providers' past performance 
although other factors were more im- 
portant, including availability of alter- 
native providers and characteristics of 
the local economy. 

• SD A performance expectations for 
service providers and how SDAs com- 
municate them to the providers 
through contract terms, monitoring, 
and related practices. Nearly all 
SDAs built performance expectations 
into both performance-based and cost- 
reimbursement contracts. SDAs also 
used many noncontractual procedures 
to ensure high performance. 
The final section describes service 
providers' responses to performance expecta- 
tions. As was the case with SDAs, service 
providers varied in the extent to which perfor- 
mance goals were given precedence over 
other organizational goals. Most providers 
could meet their performance goals without 
substantial revision to program design or 
client selection. 

SDA Management Goals: Case 
Study Findings on Competing 
SDA Objectives 

As the administrative entity responsible 
for expending JTPA funds in its local service 
delivery area, each SDA has the task of im- 
plementing a program that will further a 
variety of management goals. Exhibit 8 lists 
some of the goals that are generic to this 
management responsibility. The specific con- 
tent of the goals are shaped by a variety of 
local, State, or Federal program priorities (in- 
tended outcomes). Management practices are 
then implemented to further the specific 
goals. However, some of the management 
practices intended to further one goal also 
have unintended or indirect effects on the out- 
comes in other areas. 



Goal 1: To ensure that the planned 

mix of clients is actually served. 

In some of die case study sites, the 
planned mix of clients was based on a strong 
local policy commitment to serve particular 
target groups (for example, youth at risk of 
dropping out of school or welfare mothers). 
In other SDAs, the planned mix of clients 
was more heavily influenced by external fac- 
tors such as State priorities for serving par- 
ticular groups or Federal requirements for 
equitable service levels to dropouts or AFDC 
recipients. In a tfiird group of case study 
sites, the SDA had a policy of serving 
whoever "walks in the door" (usually in local 
areas where the SDA did not have to turn 
many people away due to a limited demand 
for JTPA services). 

Many of the SDAs in the case study 
sample were concerned about ensuring that 
the actual client mix was similar to the 
planned mix so that the final performance- 
standard levels would be close to die planned 
levels. This was not the case in all SDAs. 
Some SDAs were performing at so high a 
level in relation to the standards diat they 
were not in danger of missing any standards. 
Odier SDAs were indifferent to whether or 
not diey met the standards and did not use 
die performance standards as a front-end 
management tool but merely compared their 
end-of-year performance to the standards to 
see if they had met die standards. 

Goal 2: To ensure that the planned 
mix of services is actually provided. 
As described in Chapter X, the planned 
mix of services was influenced by a variety 
of factors, including PIC preferences, CETA 
historical patterns, service needs of die in 
dividuals in die JTPA applicant pool, and 
resource constraints and limits created by the 
cost performance standards. This was one 
area in which the Federal Government and 
die States generally took a hands-off posi- 
tion, leaving program design to local discre- 
tion. 
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Exhibits 

Examples of Management Goals for an SDA Administering the JTPA Program 

Goal 1 : To ensure that the planned mix of clients is actually served 
Goal 2: To ensure that the planned mix of services is actually provided. 
Goal 3: To ensure that each client is referred to services that are appropriate to his/her 
needs and interests. 

Goal 4: To ensure that each service provider enrolls clients that are appropriate for that 
service and have some likelihood of having a successful outcome after receiving 
the service. 

Goal 5: To ensure that costs are kept within reasonable limits. 
Goal 6: To ensure that desired performance levels are achieved. 



155 1-95 



Qualitative Analysis 



The extent to which the performance 
standards shaped the planned mix of services 
varied dramatically among SDAs. In general, 
the cost standards had the greatest effect on 
the desired service mix, although the average 
wage at placement standard also influenced 
the mix of occupational areas in which train- 
ing was offered as well as the extent to which 
direct placement or job-search assistance ser- 
vices was offered. 

Goal 3: To ensure that clients are 
referred to services that are 
appropriate to their needs and 
interests. 

This goal, a general principle of good 
management for any social service program, 
received differing amounts of emphasis 
among the case study SDAs. The achieve- 
ment of a good match between client needs 
and the range of available services depended 
on what practices the SDAs and service 
providers had devised for assessing applicant 
interests and skills and for enrolling clients in 
particular training programs. The case study 
SDAs ranged from highly centralized 
management designs, in which the respon- 
sibility for ensuring that the applicant was en- 
rolled in an appropriate program was held by 
the SDA, to very decentralized management 
designs, in which each service provider had 
control over decisions about who would be 
enrolled in what service. 

Performance standards did not directly af- 
fect SDA orientations to this goal. However, 
there were some indirect effects, based on 
whether the SDA placed a priority on meet- 
ing the needs of clients, meeting the needs of 
employers, or meeting the performance stand- 
ards. In SDAs with a priority on meeting the 
needs of clients, management practices were 
oriented toward inclusiveness in designing 
the client/service match: making sure that 
there were services available for the needs of 
different clients and making sure that each 
applicant was able to access those services 
most appropriate to his/her needs and inter- 



ests . 

In SDAs with a priority on meeting the 
needs of employers, management practices 
were oriented toward training individuals for 
the demand jobs in the local economy in 
designing the client/service match. In these 
SDAs, an effort was made to provide more 
highly skilled workers for technologically 
demanding jobs and to give employers sub- 
stantial choice in the selection of workers for 
entry-level OJT positions. 

In SDAs with a priority on meeting the 
performance standards, the client/service 
match was oriented toward enrolling in- 
dividuals who were most likely to ''succeed " 
in each particular training activity. 

Goal 4: To ensure that each service 
provider enrolls clients that are 
appropriate for that service and have 
some likelihood of having a successful 
outcome after receiving the service. 
Although all SDAs were interested in fur- 
thering this generic management goal in 
some form, the actual orientation of SDA 
management practices designed to further 
this goal varied, depending on whether they 
had a primary client orientation, an employer 
orientation, or a performance-standards orien- 
tation. The actual management practices sur- 
rounding the decision to enroll clients in a 
particular service followed the range from 
highly centralized to highly decentralized 
described under Goal 3 above. 

SDAs with a primary client orientation 
tended to design management practices to en- 
sure that providers enroll clients in need of 
the service and able to benefit from that ser- 
vice. Thus, although SDAs often permitted 
service providers to establish skills prereq- 
uisites for particular training projects, they 
would usually try to encourage providers to 
be inclusive within the category of ap- 
propriate applicants, rather than selecting 
only the easiest to serve within that category. 

SDAs with a primary employer orienta- 
tion tended to permit OJT employers to exer- 
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cise substantial discretion in selecting among 
OJT applicants and to be somewhat more 
selective in screening appropriate applicants, 
in order to increase the likelihood of placing 
trainees with employers at the conclusion of 
training. 

Finally, SDAs with a primary perfor- 
mance-standards orientation tended to assist 
their service providers in screening out the 
high-risk applicants or assigning the highest 
risk applicants to the cheapest services. 

Goal 5: To ensure that costs are kept 
within reasonable limits. 

As described in earlier chapters, SDA 
cost concerns were derived from several sour- 
ces, only one of which flowed directly from 
the performance standards. A more stringent 
cost concern for most of the case study SDAs 
was how to make a shrinking JTPA Title 
II- A allocation cover services to as many 
people as possible. This concern was 
dominant for a majority of the site visit loca- 
tions where the unemployment rate had fal- 
len since the start of JTPA programs in 
FY 84 and was causing a number of SDAs 
to reduce the average cost per terminee. 

The cost performance standards were a 
real, although in most cases secondary, con- 
cern. In some SDAs, the actual cost of provid- 
ing JTPA services did not approach the cost 
limits created by the performance standards, 
either because additional resource streams 
were used to supplement Title II-A funds in 
providing services or because the local ser- 
vice delivery mechanisms (for example, com- 
munity colleges for the delivery of classroom 
training) were extremely cost e'^ective. In 
other SDAs, the cost of providing JTPA ser- 
vices was higher (for example, in rural SDAs 
where it is difficult to achieve economies of 
scale) and it was harder to design programs 
that met the cost standards. A number of 
SDAs were trying to adjust their programs 
to meet the needs of applicant pools with 
greater basic skills deficiencies, and these 
SDAs also found it more difficult to design 



more intensive programs within the limits of 
the cost standards. Finally, in some States, 
the SDAs were highly motivated to "over- 
achieve" on the cost standards, because this 
resulted in the largest possible incentive 
award. Thus, a number of SDAs in the case 
study sample were paying far less for JTPA 
services than the cost standards themselves 
would allow and performed at levels ranging 
ftt)m 30 to 60% of the cost-per-entered- 
employment standard. 

Goal 6: To ensure that desired 
performance levels are achieved. 

The extent to which SDAs were inter- 
ested in meeting or exceeding their perfor- 
mance standards depended on whether the 
performance standards were seen as consis- 
tent or in tension with other local goals and 
on how important it was to the SDA either to 
maximize its 6% incentive awards or to "look 
good" on the standards in comparison to 
other SDAs in the State. In addition, some 
SDAs found it relatively easy to meet the 
standards without altering their desired 
program design and client goals, while other 
SDAs found it difficult and had to carefully 
tailor their program or modify their client tar- 
geting goals to meet the standards. Most 
SDAs fell somewhere between these two ex- 
tremes and used a variety of management 
practices to ensure that they could meet the 
standards while pursuing the particular 
program mix and client goals that they 
preferred. 

Development of Strategies to Achieve 
Performance Goals 

Appendix D describes the strategies that 
each of the 30 case study SDAs had 
developed in response to their individual per- 
formance goals, including modifications to 
client target groups, service designs, and 
management practices. 

As described in the previous chapters, 
most SDAs did not alter their client goals or 
desired program mix in response to the per- 
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formance standards. For these SDAs, the ef- 
fect of the standards was largely limited to 
program management practices to ensure that 
the reported outcomes would conform to the 
levels required by the standards. The 
strategies associated with increasing or 
safeguarding performance on specific stand- 
ards are described below. 

SDAs used a variety of program manage- 
ment tools to increase the likelihood that they 
would meet the entered-employment-rate 
standards . The following practices were im- 
plemented bj' one or more SDAs: 

• Selecting service providers with a 
proven track record and retaining 
only those contractors that performed 
up to expectations; 

• Using performance-based contracts to 
hold back part or all of the service 
provider's payment until the desired 
placement outcome had been reached; 

• Holding the service providers respon- 
sible for placement according to clear- 
ly stated goals in cost-reimbursement 
contracts and monitoring contractors 
frequently on their progress in meet- 
ing contract goals; 

• Retaining ultimate responsibility for 
placement at the SDA level, to ensure 
that the required placements were 
being made; 

• Introducing long assessment phases 
between the time of application and 
enrollment to ensure that those who 
were enrolled were really motivated 
to participate; and 

• Not enrolling direct-placement clients 
until they had obtained employment. 

A somewhat different set of management 
rcsponses were used by SDAs that were wor- 
ried about meeting or exceeding the average 
wage-at-placement standard . Strategies to fur- 
ther achievement on this standard included 

• Setting a specified minimum wage 



for placements using OJT, cus- 
tomized training, or classroom train- 
ing; 

• Setting specific wage goals for direct 
placements; 

• Reviewing the planned training mix 
to ensure that the wage standard 
would be met; 

• Avoiding occupational training areas - 
that would lead to low-wage jobs 
(several SDAs avoided child care and 
food service occupations; one deleted 
auto mechanics training after cars be- 
came so electronically sophisticated 
that their trainees could not get jobs 
except at service stations); 

• Making a special effort to recruit new 
employers into an OJT program that 
would offer higher paying jobs; 

• Offering service providers a bonus 
for achieving a placement above a 
given specified wage level; and 

• Taking greater care in designing OJT 
contracts. 

Management practices oriented toward 
ensuring that the cost standards were met in- 
cluded the following: 

• Including unit costs that would en- 
sure meeting the standard into the per- 
formance-based contracts of service 
providers; 

• Requiring contractors to repay inter- 
mediate payments to the SDA if cost- 
per-entered-employment goals were 
not met; 

• Seeking other sources of funding to 
supplement Title II-A funds, both as 
a way of easing the pressure on ad- 
ministrative costs and enabling them 
to dual-enroll clients; and 

• Selecting service providers with alter- 
native funding, reducing the JTPA 
costs of the services. 
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These examples illustrate that SDAs have 
adopted some management practices in 
response to specific performance-standards 
concerns. The case studies also examined in 
detail the reasons that SDAs chose enroll- 
ment practices and service-provider arrange- 
ments. The next section describes an analysis 
of the relative role of performance standards 
and other factors in how SDAs managed their 
programs. 

Practices Concerning Assessment, 
Assignment to Services, and 
Enrollment 

Assessment and Assignment to Services 

SDA program designs and management 
practices relating to client assessment, assign- 
ment to services, and enrollment can be 
oriented toward the fulfillment of several dif- 
ferent management goals, including ensuring 
that the planned client mix is achieved, ensur- 
ing that clients receive appropriate services, 
and ensuring that performance objectives are 
achieved. If management practices in this 
area are disproportionately oriented toward 
performance objectives, they can result In 
"risk avoidance" strategies that skew the 
JTPA client population toward easier-to- 
serve clients. However, if management prac- 
tices in this area are based on a "risk 
management* * strategy that attempts to 
balance performance concerns with other 
client and service goals, they can ensure the 
achievement of performance goals without 
sacrificing service to hard-to-serve clients 
and local service preferences. 

The outreach function was generally a 
function jointly exercised by SDAs and in- 
dividual service providers. Even when one or 
the other was given primary or official 
responsibility for recruitment, the dearth of 
applicants for JTPA programs in many of the 
case study SDAs sent both levels of program 
actors scrambling for as many recruits as pos- 
sible. 

Of the 28 case study SDAs for which suf- 



ficient information on recruitnent/assess- 
ment/referral practices was available, 20 
(71%) had centralized the responsibility for 
assessment and referral of applicants to a 
single source, while 8 SDAs (29%) penriitted 
each service provider to recruit, screen, and 
enroll clients for its own project. In 16 of the 
20 SDAs with the centralized assess- 
ment/referral functions, the SDA was directly 
responsible for this function. In one of the 
remaining four SDAs, the responsibility was 
passed down to a number of the local jurisdic- 
tions making up the local consortium. Final- 
ly, in three cases, the centralized function 
was administered by a service provider: in 
one case the same provider that delivered all 
the services for the SDA, in one case a spe- 
cialized intake contractor, and in one case the 
local offices of the State employment service, 
which also provided some, but not all, of the 
training services to JTPA enroUees. Even 
when recruitment and screening were central- 
ized at the SDA level, often projects serving 
special-needs groups (for example, the hand- 
icapped) weie permitted to do their own 
recruitment and screening. 

Although the two primary' service 
designs — centralized versus decentralized as- 
sessment—created different issues for SDA 
management and control, they did not, by 
themselves, affect the types of clients 
selected. Our initial hypothesis was that 
centralizing assessment and referral decisions 
would give SDAs the ability to ensure that 
hard-to-serve clients were selected for enroll- 
ment, while decentralizing screening and en- 
rollment decisions might have permitted 
service providers to select easiev-to-serve 
clients. 

The case studies suggested, however, that 
most of the SDAs that had delegated the 
authority for client screening and enrollment 
to service providers trusted the providers to 
do "sensible screening," based on their com- 
mitment to serve hard-to-serve clients. Even 
more interesting was the fact that the SDAs 
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themselves had varied motives for retaining 
centralized control over the assessment/refer- 
ral process. 

Six SDAs appeared to exercise their 
centralized assessment/referral function 
without a clear agenda either to include or to 
exclude hard-to-serve clients. Tnese SDAs 
followed previously agreed-upon criteria for 
the prerequisites necessary for referral to 
specific training programs and generally did 
not exercise discretion in making referrals 
beyond these stated criteria. 

Another eight of the SDAs with central- 
ized assessment/referral practices used these 
practices to ensure that service providers in- 
cluded hard-to-serve individuals in their 
programs. At least four of these SDAs re- 
quired their service providers to accept all 
referred clients or at least to provide written 
justification for any clients that the provider 
did not accept. Another of these SDAs said 
that they specifically useH their role in recruit- 
ment and assessment to try to reach in- 
dividuals with "marginal self-esteem and low 
motivation." 

An additional two SDAs in this category 
used their role in designing employability 
development plans (EDPs) to plan for a se- 
quence of services to individuals requiring 
basic skills remediation in addition to occupa- 
tional skills training. Several of these SDAs 
placed a great emphasis on extended assess- 
ment and evaluation as the key to achieving a 
good mix between applicant needs and the 
available services (although they also recog- 
nized the value of extended pre-enroUment 
services as a mechanism to decrease the num- 
ber of dropouts after enrollment). Centralized 
assessment/referral mechanisms permitted 
these SDAs to design and monitor delivery 
of a sequenced package of services, especial- 
ly when the appropriate services were of- 
fered by more than one service provider. 

Finally, six SDAs with centralized assess- 
ment/referral mechanisms used these 
mechanisms to further performance objec- 



tives by assisting in the recruitment, iden- 
tification, and enrollment of applicants who 
would be likely to succeed in JTPA 
programs. These SDAs tended to be similar 
to others in the case study sample in steering 
those clients with the greatest formal educa- 
tion toward classroom training. However, a 
number of these SDAs had no real service al- 
ternative to offer less employable applicants, 
who were discouraged fix)m enrolling in 
JTPA services. Several of these SDAs 
described their role as sharing with their ser- 
vice providers the responsibility for screen- 
ing out the high-risk applicants and referring 
the most likely to succeed back to the 
providers. Among these providers, subjective 
assessments of motivation were used to deter- 
mine appropriateness for services, as well as 
the more objective results of screening tests. 
In these SDAs, screening tests were more 
often used to exclude individuals from a par- 
ticular service than to indicate that they were 
appropriate for targeting with a more inten- 
sive intervention. 

A second factor that seems to have in- 
fluenced some of the SDAs that chose to 
centralize their assessment/referral prac- 
tices — but does not appear to be related to 
the interest or lack of interest in ensuring 
program access for hard-to-serve clients — is 
the fact that they want to be able to select the 
most job-ready clients into a direct job- 
search component, operated in most cases by 
the SDA. As previously discussed, many of 
these SDAs also have a commitment to serv- 
ing hard-to-serve clients with more intensive 
services. 

The findings about enrollment practices 
in the case study SDAs confirm many of the 
quantitative findings. Specifically, use of sub- 
jective judgments about whether individuals 
would complete the program were found to 
decrease significantly the enrollment of hard- 
to-serve groups. In the case studies, we find 
that SDAs trying to screen out high-risk 
clients relied on subjective judgments. In ad- 
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dition, the fact that some SDAs used their 
control of the assessment process to select 
clients likely to succeed in JTPA programs 
helps explain why the quantitative analysis 
found that the proportion of services 
provided by the SD A resulted in decreased 
service to adults on welfare. The quantitative 
analysis also found that SDAs that used basic 
skills and educational criteria served more 
hard-to-serve groups. Case study SDAs with 
strong commitments to the hard-to-serve 
groups often used extensive testing to match 
clients to appropriate training. 

Timing of Enrollment 

Another dimension of SDA management 
practices that was influenced by the existence 
of the performance standards was the 
decision about when to enroll JTPA par- 
ticipants. Case study SDAs used two types of 
practices to reduce the risk of serving hard-to- 
serve clients. The first was the delivery of ex- 
tended pre-enroUment services, used by at 
least three of the case study sites. If the pre- 
enroUment services are paid for from some 
funding source other than Title II-A funds, 
there is no regulatory difficulty with this prac- 
tice. A number of SDAs used other funding 
sources to pay for extended basic skills 
remediation prior to enrollment in Tide II-A- 
funded occupational skills training. However, 
at least one of the case study SDAs used Title 
II-A funds to deliver "assessment'' services 
that lasted 4 weeks and included basic skills 
remediation in addition to testing client inter- 
ests and aptitudes. Another SDA referred ap- 
proximately 50% of its applicants to a 3-day 
pre-enroUment assessment service operated 
by a separate contractor. 

Pre-enrollment services were attractive to 
the case study SDAs for two different 
reasons. First, if paid for from other funding 
sources, they permitted more intensive ser- 
vice interventions to fit within the cost con- 
straints created by the cost standards. 
Second, extended pre-enrollment services 
made it more likely that those who "sur- 



vived" the pre-enrollment period would be 
motivated enough to stick with the program 
until the completion of training. 

A second practice related to JTPA enroll- 
ment practices was the agreement to define 
enrollment as occurring at some point in time 
slightly after training actually started. 
Delayed definitions of enrollment were men- 
tioned in the case study writeups for 10 of the 
30 case study SDAs (33%) and the practice 
may have been even more widespread. The 
time period before official enrollment has 
been defined, by convention, as an "orienta- 
tion" period. For most SDAs utilizing this 
convention, the duration of the orientation 
period has been defined as 3 to 5 days. The 
most extreme case was an SDA that defined 
enrollment in classroom training as occurring 
after the 10th class meeting. This practice al- 
lowed service providers to "backfill" (that is, 
enroll another client as though the one who 
dropped out never existed) and collect the 
maximum amount of their contracts e/en 
though there were early dropouts. 

Slighdy different issues arose in the 
definition of enrolbnent in OJT and direct 
placement services. It appears to be nearly 
universal to enroll clients in OJT at the time 
the OJT contract begins, rather than at the 
time the project first begins to work with a 
client to locate an O.TT position. Thus, for 
OJT, individuals who do not obtain a training 
position are usually never enrolled in the 
program. This makes OJT a particularly at- 
tractive option in many SDAs because it is 
seen as a "less risky" service than others. 
Two exceptions to this practice with respect 
to OJT enrollments were observed in the case 
study SDAs. In both instances, individuals 
were enrolled at the time of referral to the 
OJT component rather than at the time the 
training position started. 

For enrollment in direct placement com- 
ponents, it is not clear what prevailing SDA 
practice is. One SDA told us that tiiey do not 
enroll individuals receiving direct placement 
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services in JTPA programs until or unless 
they obtain jobs. Othtt* SDAs that operate for- 
mal job clubs or group job-search instruction 
enroll participants before placement but 
probably not on the first day of job-search 
training* 

This practice of delayed enrollment is ef- 
fective in increasing SDAs' measured perfor- 
mance. As part of the quantitative survey, 
directors were asked the average length of 
tune individuals typically received pre-enroU- 
ment services before they were formally en- 
rolled. SDAs with longer pre-enroUment 
periods had significantly higher adult and 
welfare entered-employment rates. 

Placement: Responsibility 
and Definition 

Another management practice that varies 
across SDAs is who has responsibility for 
placing JTPA participants into jobs at the 
conclusion of training. Four types of prac- 
tices were observed. First, a number of SDAs 
pass along fall responsibility for placing 
trainees to the service providers and often en- 
sure that providers will be motivated to per- 
form by carefully structuring payment points 
in performance-based contracts. Of the 
30 case study SDAs, 10 (33%) fell into this 
category. 

Second, a number of SDAs heavily in- 
volve service providers in the placement 
process (many of these SDAs have perfor- 
mance-based contracts with their providers) 
but are also prepared to assist with place- 
ments if they have not occurred toward the 
end of the 90-day holding period. Of the 30 
case study SDAs, 12 (40%) fell into this 
category. 

Third, several SDAs utilize one service 
provider as a "primary" or "umbrella" coor- 
dinator of services to all clients and involve 
this service provider in the placement pro- 
cess for all classroom training participants, 
either as the primary job developer or as a 
backup to the skills training provider. Five 



SDAs in the case study sample (17%) fell 
into this category. 

Finally, diree SDAs (10%) took primary 
responsibility for placing all JTPA par- 
ticipants through the operation of a central- 
ized job placement unit. (One of these SDAs 
also operated all training services directly.) 

These variations in placement practices 
are not associated with variations in the ex- 
tent to which hard-to-serve clients are 
reached by the JTPA program. Instead, they 
are associated with the extent to which in- 
dividual service providers are interested in 
and/or capable of taking on the primary 
placement function. Public educational in- 
stitutions are sometimes less willing or less 
interested in being heavily involved in the 
placement process, although in several SDAs 
these institutions are successfaUy operating 
under performance-based contracts and are 
taking sole responsibility for placing trainees 
at the conclusion of training. 

One of the case study SDAs that central- 
ized the placement fanction at the SD A level 
said that it needed to retain control over 
placements to make sure it would be able to 
meet the performance standards. An added 
reason was that the SDA did not want local 
employers to be flooded with job develop- 
ment contacts from a number of its service 
providers. However, the service providers in 
this SDA felt "left out" of the placement 
process and would have liked to have been 
more involved. 

In defining "placement" in contracts with 
service providers, SDAs often required that 
placements last a certain period of time (for 
example, from 10 to 30 days), be in a train- 
ing-related field, and pay a specified wage 
rate. Placements not meeting these criteria 
were either not eligible for payment under 
performance-based contracts or paid less 
than the full fixed price. 

As a result, SDAs had a certain leeway in 
reporting placements on the JASR. Some 
rOAs took advantage of this leeway by 
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reporting additional placements for which 
they did not compensate their service 
providers. Also, it is not clear tiiat all SDAs 
reported placements on the JASR tiie same 
way. Some SDAs reported all placements 
that were permitted by tiie JASR definition 
(that is, any unsubsidized placement). Other 
SDAs had policies of "not making" certain 
types of placements (for example, placements 
in *Valk-in" jobs, such as fast food res- 
taurants, or placements in temporary jobs). In 
cases in which clients obtained these types of 
jobs, SDAs may have reported tiie outcome 
as a nonpositive termination rather tiian 
reporting a low-wage job. 

Because tiiis is one area in which SDAs 
could exercise substantial control over their 
overall performance (especially influencing 
the tradeoff betveen tiie entered employment 
rate and tiie average wage at placement), 
some SDA discretion was probably used in 
how tiiey reported outcomes for specific in- 
dividuals, particularly as tiie end of the year 
approached. 

Contracting Practices 

In our survey of all SDAs, directors were 
asked about the influence of performance 
standards on tiie selection of service 
providers. Over 70% of tiie directors rated 
performance standards as either "quite" or 
"extremely" influential concerning the selec- 
tion of service providers, 59% responded tiiat 
they were "quite" or "extremely" influential 
on tiie types of contracts used, and 64.3% 
found tiiem "quite" or '^extremely" influential 
over specific contract terms. (In all three 
cases, the most frequent response was 
"quite.") 

The case studies shed light on the inter- 
pretation of tiiese results. Among the case 
study SDAs, with few exceptions, the perfor- 
mance standards did not have great impact on 
tiie initial selection of service providers be- 
cause most of these SDAs have been 
retrenching tiieir programs in response to con- 



tinuing funding declines and tiius have littie 
opportunity to consider taking on new 
providers. The standards' influence on 
provider retention is more evident, and tiiis in 
fact appears to be what SDA respondents are 
referring to when they speak of provider 
selection. 

Witiiin tiie case study SDAs, tiie greatest 
impact that tiie performance standards ex- 
erted on contracting practices was on tiie 
specific terms of individual contracts, both 
performance-based and cost-reimbursement, 
as well as on tiie increased prevalence of per- 
formance-based contracts. But it is also im- 
portant to note tiiai these impacts on 
selection, retention, and terms are generally 
tempered by tiie influences of otiier factors, 
such as tiie nature of tiie local economy, tiie 
size of the SDA's JTPA allocation, whetiier 
tiiis funding is expanding or contracting, and 
the 15% limitation on administrative expendi- 
tures. 

Selection and Retention of Contractors 

Key Contextual Factors. Two factors 
were of central importance in shaping the 
ways in which tiie JTPA performance stand- 
ards influenced the SDAs' decisions about 
service-provider selection and retention. One 
was tiie SDAs' tendency to treat contractors 
as belonging either to an essential "core" of 
service providers or to a more peripheral, 
more marginal, and tiius more dispensable 
set of providers. The otiier was tiiat most of 
tiiese SDAs had been facing declines in tiieir 
JTPA allocations over the past several years. 

Most of tiie case study SDAs had an ex- 
tremely stable core of providers witii a long 
working relationship witii tiie SDA and often 
with the CETA prime sponsor before it. The 
core typically included one or more public 
educational institutions and one or more 
CBOs or nonprofit organizations, which 
togetiier botii accounted for tiie mainstream 
of the SDA's services and clients and usually 
provided youtii services as well, enabling tiie 



203 



Qualitative Analysis 



SDA to meet its youth expenditure require- 
ment. If the SDA*s allocation was large 
enough, there would also be one or more 
providers outside the core. These were often 
smaller providers, which might provide train- 
ing in a more specialized occupation or serve 
a more specialized client group (such as the 
developmentaily disabled), and they were 
likely to have less of a history with the SDA. 

This configuration typified virtually all 
the case study SDAs that had small alloca- 
tions or served rural areas and that also did 
not conduct all activities themselves. Even 
the few SDAs that confined their coniracting 
to individual referrals typically relied on a 
small core of contractors, although in these 
SDAs the core often included proprietary 
trainers as well as CBOs and public educa- 
tional institutions. 

Besides history, core providers often 
shared an overall orientation and a close 
working relationship with the SDA. Usually 
the PIC and SDA were reasonably pleased, 
even enthusiastic, about the core providers* 
past performance; they had often had years to 
work out any problems. Core providers were 
hard to give up — precisely because they 
delivered "core" services — and hard to 
replace, although the SDA often had to 
"shave" their contracts as JTPA funding 
declined.* 

The qualitative finding that core 
providers and the SDA usually shared com- 
mon program objectives helps explain the 
quantitative finding that the type of provider 
had little impact on the types of clients en- 
rolled in JTPA programs. 

At least 22 of the case study SDAs (73%) 
were experiencing declines in JTPA alloca- 



tion. In this environment, there was usually 
little practical opportunity to select complete- 
ly new providers, except at the margin or in a 
situation of major program restructuring.** 
Instead, the typical decision facing the SDA 
was how to cUstribute funding cuts among 
providers, and in this situation the core/ 
periphery distinction was critical. As men- 
tioned previously, more marginal providers 
were usually much more vulnerable to 
having their contracts discontinued or sustain- 
ing major cuts, and the competition for sur- 
vival was usually fiercest among them. Heiie, 
past performance could be a telling factor. 

Impact of the Performance Standards 
on Choices among Providers. Generally, 
those SDAs that v 'ere oriented toward ex- 
ceeding their performance standards tended 
to be more demanding about bidders' ability 
to demonstrate previous success with the 
SDA's clientele. This tendency worked to the 
advantage of core providers, local providers 
(because there was less uncertainty over the 
adaptability of their program to the local set- 
ting), and providers with established 
programs. It penalized new organizations, 
providers from outside the SDA, and innova- 
tive or experimental approaches. Consequent- 
ly, it had an overall conservative influence 
on the selection of service providers. 

With respect to specific standards, SDAs 
that were particularly concerned about their 
overall performance on any of the entered- 
employment-rate standards naturally favored 
service providers with strong placement 
records. If average wage was a concern, the 
SDA might drop a contract (or proportion of 
a contract) for training in a particular occupa- 
tion with an especially low wage rate. SDAs 



*Our site visits turned up one notable exception to this general pattern: an SDA reallocated a youth program 
from the public school system— a core provider— to a local nonprofit organization because of displeasure with the 
quality and performance of the former provider. 

**Five SDAs had experienced a significant reorganization since 1986. One had no conu-actors. two were loo 
small to have more than a handful, and in two, the restructuring appears to have been prompted by factors unrelated 
to performance. 
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for which cost-per-entered-employment 
standard posed a problem tended to pressure 
providers to lower their unit costs or to give 
priority to providers who could leverage out- 
side funding."^ 

Overall the performance standards played 
a significant part in determining which ser- 
vice providers were retained as an SDA's al- 
location declined, especially among the more 
peripheral providers. They also helped deter- 
mine how cuts were shared among remaining 
providers, including those within the core. As 
a result, even in the absence of formal com- 
petitive bidding, many core providers as well 
as marginal providers perceived that there 
was strong competition for the SDA*s avail- 
able dollars from year to year and were atten- 
tive to their performance even when they had 
cost-reimbursement contracts.** 

Two-thirds of the case study SDAs said 
that underperformance had been a major fac- 
tor when they had reduced contracts over the 
past few years, and five cited one or more 
contractors whom they had dropped for poor 
performance. The most conmion problems 
were disappointing placement rates and ex- 
cessive costs. Among the survivors, public 
schools and commr lity colleges predomin- 
ated, followed by CBOs and other nonprofit 
organizations and public agencies. Our case 
study sample of 87 providers serving the 30 
SDAs Triplicated this general pattern: 23 
(26%) wer^ public educational institutions, 
20 (23%) were CBOs, 3 (3%) were nonprofit 
educational institutions and 12 (14%) were 
other nonprofit organizations, 9 (10%) were 
the SDAs themselves, 12 (14%) were other 



public agencies, 4 (5%) were proprietary 
schools, and 4 (5%) were other types of for- 
profit organizations. 

Other Factors Influencing Provider 
Sdection and Retention. A number of addi- 
tional factors influenced provider selection 
and retention, often mere strongly than the 
performance standards per se. Primary 
among them were the r cture of the supply of 
providers available to the SD A, the charac- 
teristics of the local economy, the amount 
and categories of services that the SDA con- 
tracted out, the goals and philosophy of the 
PIC/SDA, the demographic service require- 
ments imposed on the SDA, and the type of 
provider. 

Half of the case study sites were rural 
SDAs with limited local supply of alternative 
providers. Rural SDAs were also generally 
least willing to go outside their aiea for 
providers. Often, especially in the most far- 
flung and sparsely populated aieas, a rural 
SDA already faced considerable difficulty in 
connecting potential clients with existing 
providers, due to long distances and inade- 
quate transportation systems. An SDA in 
these circumstances had few alternative 
providers to turn to if it discontinued a con- 
tract for poor performance, so it had a greater 
incentive to try to work with the provider to 
cure performance problems. 

Some SDAs ruled out certain types of 
training providers or made only individual 
referrals to them because the occupations for 
which they offered training were not in suffi- 
cient demand within the local economy. 
Other providers were omitted because they 



♦This capacity to match JTPA funding tends to counter the ••periphery" disadvantage in the case of programs 
that can augment a JTPA contract with vocational rehabilitation fimds. A combination of this additional funding 
plus PIC or SDA philosophy may have produced a substantial number of contracts serving the developmentally dis- 
abled or other handicapped groups among the case study SDAs. 

**Even excellent performance might not be enough tc guarantee a provider's survival if the funding cuts were 
severe enough. One SDA was planning to discontinue one cf its more peripheral contracts even though the project 
had exceeded all requirements and established a record of cost- effectiveness. The problem was that, although cost- 
effective and successful, this nrogram entailed high costs that the SDA felt it could no longer afford once its alloca- 
tion sustained another significant reduction. 
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trained for low-paying occupations, such as 
child care worker. In both cases, perfor- 
mance standards probably played a role in 
the decision: the entered-employment rate in 
the first instance, the average wage standard 
in the second. But equally important seemed 
to be overall PIC or SDA judgments about 
the types of jobs that were "right" for local 
clients. 

SDA client goals and service mix 
decisions also had a strong input on the 
choice of provider. For example, some PICs 
placed a strong emphasis on dropout preven- 
tion or on providing remedial services; this 
tended to favor the schools and "alternative" 
programs that targeted dropout-prone youth 
or clients in need of basic skills remediation. 
The fairly frequent targeting of a certain por- 
tion of the SDA's budget for the developmen- 
tally disabled, the mentally ill, or other 
"special" client groups generally favored 
those providers who specialized in serving 
such groups. Some SDAs showed clear 
preferences for different types of providers. 
For example, prevailing attitudes for or 
against proprietary providers influenced the 
distribution of funding between proprietary 
and all other providers, while an emphasis on 
taking full advantage of the resources avail- 
able through the public schools benefited 
these institutions. 

One additional factor that could dffect 
choices among service providers was local 
politics. In most of the case study SDAs, this 
was not a significant factor. However, in four 
of the largest SDAs, respondents did offer 
comments on the relationship between 
politics and the performance standards as in- 
fluences on provider selection. In two of 
them, the observers judged that incorporation 
of the performance standards had significant- 
ly reduced the influence of politics on 
decisions about whether to retain providers 
and about their funding levels. In another, 
staff believed that such decisions were often 
still subject to political considerations and 



lobbying but were hopeful tiiat the standards 
were gradually making the decisionmaking 
more strictly "objective." In tiie fourth of 
tiiese SDAs, observers characterized the 
selection process as heavily politicized. 
However, they also said tiiat political in- 
fluence concentrated on decisions of whether 
to retain or drop providers. When the 
decision concerned the depth of a provider's 
funding cutback, on tiie other hand, tiie per- 
formance standards were the dominant factor. 

Process Through Which Contractors 
Are Chosen. Among tiie case study SDAs, 
22 (73%) conducted a formal Request for 
Proposals (RFP) process in selecting 
providers for at least a portion of their ser- 
vice program. Nineteen of tiiese SDAs issue 
open, nonspecific RFPs and depend on in- 
dividual service providers to propose training 
programs to tiie SDA for funding. At least 
six SDAs in tiie study sample indicated tiiat 
they depended on the specific proposals 
made by service providers to shape their 
program design, which can lead to innova- 
tion (for example, a provider introduced the 
concept of open-entry, open-exit training to 
one SDA), but can also lead to a large, hap- 
hazard program (one SDA had 85 contracts, 
mostiy small ones with CbOs, which were 
difficult to monitor). Providers were par- 
ticularly influential in proposing the design 
for special projects serving particular target 
groups, such as the handicapped, or of- 
fenders, or in-school youth. However, an 
open approach can also result in some groups 
being unserved if no service providers plan 
to include them in their programs. 

Of the remaining eight SDAs (27%), one 
had no outside service providers, two did al- 
most everything themselves and relied on in- 
dividual referrals for any externally provided 
training, and five said that they "no longer" 
employed RFPs, at least as long as existing 
providers' performance remained satisfac- 
tory. Instead, they either renegotiated con- 
tract terms with current contractors before a 
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new funding cycle began or mapped out the 
services that they wanted contractors to 
deliver and then asked a few good providers 
to bid on their design. 

In at least 9 of the 22 SDAs with RPPs, 
respondents characterized the REP as only 
"technically" open or as applying only to 
more marginal providers; and in virtually all 
of the 27 SDAs doing significant amounts of 
contracting, there was considerable stability 
in the "cast" of contractors (especially at the 
core). However, even in the SDAs without a 
RFP procei^s or in which it was only a for- 
mality, there was genuine competition. The 
negotiations stage was when the real changes 
were made and was frequently the scene of 
very hard bargaining over costs and perfor- 
mance goals. 

Contract Terms 

Types of Contracts Used. Among the 29 
case study SDAs that did any contracting, 20 
(69%) had at least some current performance- 
based contracts, and several planned to ex- 
pand their use. Of the nine SDAs (31%) 
without performance-based contracts, two 
had retreated from earlier failures with such 
contracts, and one was an SDA that was like- 
ly to begin some performance-based contract- 
ing. In addition, two SDAs had some hybrid 
contracts, cost reimbursable but on a 
capitated basis. Public educational institu- 
tions and otSr : public agencies were most 
likely to remain on cost-reimbursement con- 
tracts, although for-profit providers were 
especially likely to be on performance-based 
terms. 

Although performance standards and per- 
formance-based contracts are often men- 
tioned in tandem, the standards are not the 
only factor in the spread of the performance- 
based format. Among the case study SDAs, 
the 15% administrative limit, especially in 
the context of declining Title II-A alloca- 
tions, was at least as important as an SDA's 
desire to gain tighter control over providers' 



performance. One of the case study SDAs 
was considering converting at least some of 
its contracts to a performance basis precisely 
because its allocation was falling to a level in 
which it doubted whether it could stay within 
the 15% limit otherwise. The continuity of 
CETA staff did not seem to inhibit the adop- 
tion of performance-based contracts, and at 
least one SDA that had been a prime sponsor 
reported using them under CETA. 

Typical Contract Terms. In perfor- 
mance-based contracts, typical contract terms 
that reflect the influence of an SDA's perfor- 
mance standards include 

• Minimum or maximum numbers of 
enroUees, 

• Detailed screening criteria or a mini- 
mum proportion of client referrals 
that the provider is obliged to accept, 

• Minimum percentages for specific 
demographic groups (most typically, 
youth), 

• Degree of specificity about the 
provider's cuiriculum, 

• Specification of the average, maxi- 
mum, or (more occasionally) mini- 
mum duration of service, 

• Goal for the number or rate of place- 
ments and the specific definition of 
their required characteristics, 

• Percentage of payments to be with- 
held until placement (or placement 
above a specified wage or meeting 
other criteria of suitability) or the 
amount of bonuses available for extra 
or high-wage placements, 

• Nature and distribution of interim pay- 
ment points, if any, and 

• Total amount authorized under the 
contract and the unit cost. 

Cost-reimbursement contracts are similar 
in many respects. However, in general, 
provisions concerning the levels and charac- 
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teristics of enrollments and placements (or 
training completers, since cost-reimburse- 
ment contractors are less likely to have for- 
mal placement responsibility) are framed as 
goals rather than requirements. In the few ex- 
amples in which the contract language ap- 
peared to express requirements, the contract 
^•'^tually provided no means to enforce them, 
Ir; addition, cost-reimbursement contracts did 
r;ot specify unit cost (except in the two sets 
of "hybrid" capitated training contracts men- 
tioned above) and did not provide for place- 
ment holdbacks. 

Influence of Performance Standards 
on Contract Terms. In the case study 
SDAs, the great majority of contracts built 
the SDA's standards into the contract terms 
of its providers, as either requirements or 
goals for performance-based and cost-reim- 
bursement contracts alike. 

In approximately half of the SDAs, con- 
tracts built in uniform requirements of either 
the SDA's standards or the "standards-plus-a- 
cushion" that protected the SDA in the event 
of disappointing performance by a contrac- 
tor. The other half employed a more compli- 
cated "jigsaw" approach of letting key 
contract terms vary among their providers 
but taking care that, aggregating across the 
contracts, they met or exceeded the SDA's 
own standards. Only one SDA had estab- 
lished a mechanism to vary terms according 
to client characteristics using the DOL adjust- 
ment methodology. 

Concerning enrollment levels, most of 
the case study SDAs that had class-size ser- 
vice contracts at least established enrollment 
goals; eight specified minimum numbers (all 
of them had trouble recruiting clients), while 
two imposed maximum levels. There was no 
clear pattern in requirements covering client 
characteristics; many specified only Title II- 
A eligibility, although a number specified 
youth participation levels. Similarly, SDAs 
and individual contracts varied in the 
specificity of their description of curriculum 



and other services. The great majority 
specified duration, usually as an average and 
a maximum; one, reflecting a strong conraiit- 
ment to quality, also specified the minimum 
time in training. 

In placement-oriented contracts, the typi- 
cal requirement or goal for the percentage of 
clients to be placed ranged between 60 and 
80%, although a few contracts covering hand- 
icapped clients allowed somewhat lower 
placement rates. Eight SDAs had contracts 
specifying placement holdbacks exceeding 
50% (several of them allowed no interim paj - 
ments), 10 held back fix)m 20 to 50%, and 5 
held back 20% or less. (Because about half 
of the SDAs mixed types of contracts, includ- 
ing cor/L reimbursement and performance 
based, these figures overlap.) In a contrast- 
ing approach, one SDA awarded a $100 
bonus for every placement. Among the 
SDAs using performance-based contracts 
with low to moderate placement holdbacks, 
the contract typically provided from 3 to 5 in- 
terim payment points; in SDAs facing recruit- 
ment difficulties, the first payment point 
might be enrollment with the contractor. 

The high-placement-holdback contracts 
illustrate one extreme of a spectrum of SDA 
approaches to managing and allocating the 
risk entailed in trying to meet or exceed their 
performance standards. In establishing a high 
placement holdback, the SDA in effect as- 
signed a maximum share of the risk to the 
provider. Other SDAs adopted a "risk-shar- 
ing" strategy by using less extreme place- 
ment holdbacks and providing more 
generous interim payments in deference to 
their providers' cash flow needs. These 
SDAs were heavy users of the noncontrac- 
tual control mechanisms discussed in the fol- 
lowing section. 

Virtually all the case study SDAs defined 
placements as full time (or somewhere above 
30 hours a week) and unsuosidized; at least 
20 specified a target or required wage, 12 re- 
quired a training-related placement, and at 
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least 9 specified job retention (varying be- 
tween 2 weeks and 1 month). Only a handful 
paid overt bonuses for high-paying or other 
"quality" placements. However, in several of 
the SD As, 5 or 10% "holdbacks" for such 
characteristics amounted, in effect, to 
bonuses. Three of the SDAs incorporated 
"wage steps" in their OJT contracts to bring 
clients to or above the SDA's wage standard 
by the time that the employer's subsidy 
ended. 

Influence of Other Factors on Contract 
Terms. The relationship between contract 
terms and the SDA's performance standards 
was affected by several factors. Half of the 
case study SDAs were having difficulty 
recruiting clients for their programs (par- 
ticularly youth), and half of these also faced 
a limited provider supply. In areas where 
clients were in short supply, because of a 
healthy or improving local economy, those 
available were generally seen as harder to 
serve and less marketable than the average 
client several years earlier. In these areas, 
placement goals were often set at more 
modest levels, and contract terms were less 
exacting than in the five SDAs where 
providers could pick and choose among ap- 
plicants.* 

However, if the SDA also faced a drastic 
reduction in its Title II- A allocation, as it 
often did when the economy was improving, 
a desire to spread available resources as far 
as possible — even more than the cost stand- 
ards — led to constant and strong pressure on 
providers to reduce costs. In addition, in 
those SDAs that attempted to earn all avail- 
able 6% funds, the SDA tended to push not 
only for lower costs but also for higher place- 
ment and wage rates. 

There were two primary forces counter- 
ing this pattern: the availability of non-JTPA 
resources (either administered by the SDA or 



accessible through a service provider) and a 
"client oriented" FIC or SDA in favor of serv- 
ing a variety of clienic. SDAs that gave 
preference to CBO2 as providers to assure 
that various hani-to-serve groups were served 
were likely to offer them relatively generous 
interim payment points, if they subjected 
these providers to performance-based con- 
tracting at all. 

In a number of the SDAs, programs serv- 
ing physically, mentally, or developmentally 
handicapped clients appeared to enjoy special 
protection (considering their typicdly lower 
placement and wage rates and higher costs), 
although they were not all spared significant 
funding cuts. Although some case study 
respondents mentioned the PIC's sense of 
"what is right" or the local program's com- 
munity orientation, the fact that these 
programs generally incorporated vocational 
rehabilitation funds also favored them, espe- 
cially through reducing their apparent cost 
(in JTPA dollars). Similarly, State or local 
dollars targeted to welfare recipients, youth, 
substance abusers, or others also enhanced 
the prospects that programs serving these 
groups would receive SDA funding. 

Limitations on Contract Terms as 
Means for Promoting Performance. SDA 
contract terms should not be understood as 
simple outgrowths or reflections of the JTPA 
performance standards. Clearly, enrollment 
and placement goals, demographic quotas, 
screening criteria, cost limits, and other fea- 
tures of service-provider contracts predate 
JTPA programs, as does the use of perfor- 
mance-based contracts in several of the case 
study SDAs. Moreover, even where the form 
of performance-based contracts is a recent 
development, their key terms do not always 
represent a major change. Where they are 
now in use for long-time providers, they 
often essentially continue the mutual expecta- 



*These were also the SDAs most likely to specify minimum enrollments in contracts, whereas those specifying 
maximum enrollment levels typically faced an oversupply of eligible applicants. 
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tion J and the level of performance that 
characterized prior years. 

Performance-based contracts did not 
guarantee performance success — one SDA 
that relied on them exclusively missed 
several of its standards — and their absence 
did not necessarily spell trouble with the per- 
formance standards. The reason is that SDAs 
have a number of other potentially effective 
mechanisms at their disposal for controlling 
and motivating provider performance. 

Noncontractual Control Mechanisms 

Besides specifying and enforcing con- 
tract terms, SDAs can influence provider per- 
formance through the following means: 

• Organization of intake, assessment, 
and referral of clients, as discussed 
previously; 

• Degree of latitude providers are al- 
lowed to refuse applicants or refer- 
rals, including specific criteria 
governing acceptance or justifying 
refusal; 

• Adoption of a case management ap- 
proach aimed at not letting accepted 
clients "fall through the cracks" and 
ensuring that they receive the most 
suitable services and make satisfac- 
tory progress; 

• Use of computerized spreadsheets to 
update actual performance both for 
the SDA as a whole and for in- 
dividual providers; 

• Careful, frequent monitoring of 
provider activities, both formal and 
informal; 

• Close communications and frequent 
feedback between SDA staff and 
providers; 

• Opportunity for fine-tuning as the 
program year advances; 



• Technical assistance to providers as 
needed; and 

• Long experience with each other, a 
highly evolved service design, and a 
combination of trust and a close work- 
ing relationship. 

Another powerful means is the funding 
carrot: providers want to get a contract "the 
next time around." 

The most commonly used means of con- 
trolling service providers was monitoring per- 
formance through MIS data, used by at least 
21 SDAs. The frequency of the monitoring 
reports varied from quarterly to daily; most 
SDAs produced monthly updates, and 
several gave these heightened attention 
during the latter I^alf of the program year. At 
least 16 of the SDAs employed frequent 
monitoring visits and contacts, both formal 
and informal— in some, SDA staff visited 
programs on virtually a daily basis — and 
fostered close communications between 
providers and staff. At least 10 provided for 
fine-tuning among programs and midcourse 
corrections within programs. 

At least 12 SDAs had long histories with 
most of their providers, had worked out suc- 
cessful and adaptable service designs, shared 
goals and orientations, and had a collabora- 
tive working relationship. Generally, these 
were successful performers, at least on their 
own terms; a few in depressed rural areas 
were content to let average wage slip, and 
one consciously put more emphasis on inten- 
sive services rather than on achieving its cost 
standard. 

Not surprisingly, all the case study 
providers paid attention to "next time," their 
future funding prospects, although some 
were more anxious about it than others. A 
few said that they were getting squeezed so 
tighdy that they were looking for ways to 
decrease their dependency on JTPA funding. 
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and there were occasional reports of 
providers that had dropped out of the local 
competition; but most still wanted to stay in 
the game.* 

One SDA with performance problems 
and with only a limited set of providers had 
recently engaged in a concentrated campaign 
to recruit additional providers, doubling its 
contractor pool. Although the SDA's objec- 
tive was to gain more freedom to retain high- 
risk programs while improving its record on 
the performance standards, there was also the 
possibility that it might replace some current 
contractors if they continued to have trouble. 
Within the case study sample, however, this 
was an unusual approach; more SDAs were 
likely to confront the unpleasant prospect of 
being compelled to drop or severely prune a 
valued service program. 

How Contract Terms and 
Related Practices Affect Service- 
Provider Behavior 

Most service-provider respondents did 
not have a complete understanding of the per- 
formance-standards system as it applied to 
the SDA as a whole. Service providers, 
however, were very aware of the perfor- 
mance expectations placed upon their project 
by the payment terms of their contracts as 
well as by additional performance goals and 
objectives communicated to them by the 
SDA. In certain circumstances, as when a par- 
ticular service provider was represented on 
the PIC, service-provider staF yere also 
aware of how their particular performance ex- 
pectations fit into the larger framework of 
SDA level performance standards, but this 
was the exception rather than the rule. 

As was the case for SDAs, service 
providers varied in the extent to which they 
oriented their program-design decisions 
toward meeting their performance expecta- 



tions versus meeting other organizational 
goals (for example, client priorities and ser- 
vice goals). Li most cases, there was enough 
room to maneuver so that service providers 
did not have to dramatically redesign their 
training program or client selection proce- 
dures to meet their contract obligations. Con- 
trary to our expectations, we did not find that 
SDAs usually played the role of "protecting 
the level of service to hard-to-serve clients" 
or that service providers usually tried to "get 
away" with serving easier clients or provid- 
ing less expensive services. Rather, the 
qualitative analysis found that most service 
providers — whether they were nonprofit com- 
munity-based organizations, public education- 
al institutions, or for-profit enterprises — did 
have a commitment to serving participants 
who needed and would be able to benefit 
from JTPA services. 

The qualitative analysis did not find any 
evidence that performance-based contracts, 
per se, constrained service-provider choices 
or promoted "risk avoidance" behavior any 
more than cost-reimbursement contracts. 
However, contract terms that took into ac- 
count the particular types of service being 
provided and the types of clients being 
served by each provider generally made it 
easier for service providers to implement 
their desired training strategy than contract 
terms that were standardized across all ser- 
vice providers in an SDA. These results arc 
very consistent with the quantitative findings 
that use of performance-based contracts did 
not affect the types of clients served but that 
varying the terms of contracts did increase 
service to some hard-to-serve groups. 

At the extreme, pressure to perform at 
high levels (under either performance-based 
contracts or cost-reimb'orsement contracts) 
did cause greater caution in client selection 
and in service design among service 



*0f course, it is important to note that study respondents were current contractors, and the story might have 
been different had we talked to contractors that had decided not to reapply for JTPA funds. 
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providers. Particularly influential were pres- 
sures to achieve: high wage rates , particular- 
ly contract terms that specified, rainiihum 
wage rates required for payment, as opposed 
to average wage performance expectations; 
high placement rates (for example, 80% or 
higher); or high portions of payment reserved 
for achievement of placement outcomes , m 
which the service provider would risk its 
ability to earn its full contract amount if it 
failed to achieve placement outcomes for all 
or nearly all its clients. Faced with the choice 
of actually losing money if they failed to 

choose clients carefuUyrtnost service 

providers chose carefully. 

Conclusions 

SDA management practices are designed 
to further a variety of management goals, in- 
cluding ensuring (1) that the planned mix of 
clients is actually served, (2) that the planned 
mix of services is actually provided, (3) that 
each client is referred to appropriate services, 
(4) that service providers enroll clients that 
are appropriate for that service and have a 
reasonable likelihood of success, (5) that 
costs are kept within reasonable limits, and 
(6) that the desired performance levels are 
achieved. 

To further their management goals, 
SD As instituted a number of practices for en- 
rollment, assignment to services, placement, 
and selection of service providers. Some of 
these practices grew out of their approach to 
the performance standards. 

Most of the SDAs visited maintained 
centralized control over the outreach, assess- 
ment, and referral of clients to service com- 
ponents, with a smaller number leaving 
responsibility for these services to service 
providers. Contrary to expectations, the ex- 
tent of screening out hard-to-serve clients 
was not correlated with the extent of SDA 
control over enrollment. Some SDAs with 
centralized enrollment were very cautious 
about enrolling clients, and others with 



decentralized enrollment served large num- 
bers of hard-to-serve individuals. SDA and 
service-provider goals about serving the hard 
to serve seemed to be more important than 
the control of enrollment 

One area in which SDAs protected them- 
selves against the risk of serving hard-to- 
serve individuals was in the timing of 
enrollment. SDAs used two mechanisms to 
delay enrollment until after some services 
had been delivered to avoid having to in- 
clude early dropouts in their statistics. Some 
SDAs offered extensive pre-enrollment ser- 
vices thaHncluded remediation and assess- 
ment. Others referred clients to classroom 
training but defined enrollment as some time 
after the start of actual services, thus allow- 
ing service providers the chance of serving 
the maximum number of clients allowed in 
their contracts. Likewise, OJT clients were 
usually not enrolled until they had actually 
started working. Both of these practices 
removed early dropouts from the 
denominator when calculating entered- 
employment rates, thus improving perfor- 
mance on those standards. Statistical analysis 
confirms that SDAs with longer pre-enroll- 
ment periods have significantiy higher 
measured entered-employ ment rates. 

Some SDAs gave the primary respon- 
sibility for placing clients to their service 
providers, and others maintained this func- 
tion at the SDA level. When the service 
providers had performance-based contracts, 
they were often not paid the holdback for 
placement if the jobs were not training re- 
lated, did not last a specified length of time, 
or did not meet a specified wage. However, 
SDAs could report these placements on the 
JASR, thus improving their entered employ- 
ment rates and lowering costs. 

Two factors shaped the influence that the 
JTPA performance standards exerted on the 
case study SDAs' choice of service 
providers. First, the great majority of them 
had been facing declines in their JTPA alloca- 
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tions over the past several years. This en- 
vironment offered httle practical opportunity 
to select coiripletely new providers; instead, 
the typical decision facmg the SDA was how 
to distribute funding cuts among providers. 
In this situation the second factor became 
critical: the SDAs' tendency to distinguish be- 
tween a stable inner core of service providers 
and those that were more peripheral, and con- 
sequently more dispensable. Providers on the 
periphery were more vulnerable both to 
serious funding cuts and to having their 
JTPA contracts completely discontinued — 
and their performance on the entered-employ- 
ment rate and cost standards often figured 
prominently in the decision. 

A number of additional factors influenced 
provider selection and retention, often more 
strongly than the performance standards per 
se. Primary among them were the supply of 
alternative service providers available to the 
SDA, the characteristics of the local 
economy, the amount and categories of ser- 
vices that the SDA contracted out, and the 
goals and philosophy of the PIC and SDA. 

Among the 29 case study SDAs that did 
any contracting, 20 (69%) had at least some 
current pftformance-based contracts, and 
several planned, to expand their use; and one 
of the SDAs without any current perfor- 
mance-based contracts was likely to convert 
to this format in FY 88. Although the perfor- 
mance standards were a factor in the spread 
of performance-based contracting among the 
case study SDAs, the 15% administrative 
limit was at least as important as an SDA's 
desire to gain tightw control over providers* 
performance. This was especially true in the 
context of declining JTPA allocations. 

Among both performance-based and cost- 
reimbursement contracts, the great majority 
buihan SDA*s standards into the contract 
terms of its providers as either requirements 
or goals. In about half of the case study 
SDAs, contracts built in the SDA's standards 
(usually with a "cushion") directly. Most of 



the other half employed a more complicated 
"jigsaw" approach of letting key contract 
terms vary among their providers but taking 
care that, aggregating across the standards, 
they met or exceeded the SDA's own stand- 
ards. 

Despite the evident influence of the per- 
formance standards on service-provider con- 
tracts, contract terms should not be under- 
stood as simple outgrowths or reflections of 
the standards. Many of the key features of 
these contracts predate JTPA, as does the use 
of performance-based contracts in several of 
the case study SDAs. Among the case study 
SDAs, the relationship between contract 
terms and the SDA*s performance standards 
was affected by several factors. These in- 
cluded how difficult a given SDA found it to 
recruit and retain clients, the relative 
employability of available clients, the 
provider supply, the status and direction of 
the SDA*s allocation, its access to non-JTPA 
funding, and the philosophy and goals of the 
PIC and the SDA staff Even when the form 
of performance-based contracts is a recent 
development, they often essentially continue 
the mutual expectations that an SDA and a 
provider had developed in earlier years. 

The placement rates required in place- 
ment-oriented contracts typically ranged be- 
tween 60 and 80%. Eight case study SDAs 
had contracts specifying placement 
holdbacks exceeding 50%, while 10 had con- 
tracts holding back between 20 and 50% 
(there is overlap between the two SDA 
groups). This practice assigned a maximum 
share of the risk associated with the perfor- 
mance standards to the service provider, 
SDAs that, in contrast, held back littie or 
nothing for placements relied more heavily 
on noncontractual means of managing their 
providers' performance. Among these, the 
most common were regular periodic monitor- 
ing of performance using MIS data; frequent 
visits and contacts; and routine provision for 
midcourse corrections. 
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As was the case for SDAs, service 
providers varied in the extent to which they 
oriented their program-design decisions 
toward meeting their performance expecta- 
tions versus meeting other organizational 
goals. In most cases, there was enough room 
to maneuver that service providers did not 
have to dramatically redesign their training 
program or client selection procedures to 
meet their contract obligations. At the ex- 



treme, pressure to perform at high levels 
(under either performance-based contracts or 
cost-reimbursement contracts) did cause 
greater caution in client selection and in ser- 
vice design among service providers. Par- 
ticularly influential were pressures to achieve 
high wage rates : high placement rates ; or 
high portions of payment reserved for 
achievement of placement outcomes . 
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Introduction 

In the 30 SDAs visited as part of this 
study, we observed a wide variety of overall 
responses to the performance-standards sys- 
tem. In many ways, the effects of the perfor- 
mance standards on clients served, types of 
services provided, and SDA management 
practices seem to be conditioned by the ways 
in which key actors at the SDA level view 
the standards. Some SDAs saw standards as 
being consistent with the goals of the JTPA 
program and with their own locally estab- 
lished goals. Often, standards were used as 
an integral planning and management tool. 
Other SDAs saw performance standards as 
being in tension with other objectives. Still 
others viewed standards as being secondary 
to other program goals, and thus, perfor- 
mance standards were largely ignored in 
making management decisions. In some 
SDAs, different actors viewed the standards 
differently; in others, the reactions differed 



depending on the specific measure under con- 
sideration. 

Case study respondents at the SDA level 
were asked several questions related to their 
perceptions of the performance-standards sys- 
tem. First, they were asked their opinion of 
the adequacy and appropriateness of the 
standards system (as well as the actual 
measures used) as a reflection of JTPA 
program goals. Related questions were about 
the levels at which the standards are set, the 
reactions to the DOL optional adjustment 
model, and the anticipated effects of the new 
standards to be implemented in PY 88. Case 
study respondents at the service-provider 
level were asked about their familiarity with 
the performance-standards system and their 
reactions to the performance expectations 
placed on them by the SDA. 

This chapter reviews SDA respondents' 
reactions to the above set of questions. It il- 
lustrates the extent of diversity in overall 
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views of the standards, reflecting the diver- 
sity in operating environments and manage- 
ment approaches that shape the operation of 
the JTPA program at the local level. 

Overview of Chapter 
and Key Findings 

In summarizing SDA reactions to the per- 
formance standards, we first describe SDA 
opinions about the extent to which the stand- 
ards are an adequate reflection of JTPA 
program goals. The next section describes 
SDA opinions about the levels at which the 
performance standards are set. It is followed 
by a section on SDA reactions to the DOL ad- 
justment model. Next, we summarize SDA 
responses about the need for additional tech- 
nical assistance on the performance stand- 
ards. Finally, we summarize SDA responses 
to questions about data concerns and report- 
ing concerns and SDA responses to the new 
standards implemented for the first time in 
PY88. 

SDAs expressed differing positions on 
the standards. About one-third of the SDAs 
visited expressed general support for the per- 
formance standards' reinforcement of 
program accountability and thought that the 
standards were generally useful management 
tools for the JTPA system. About one-fourth 
of the SDAs visited gave the performance 
standards mixed or conflicting reviews. 
About one-third of the SDAs were generally 
neutral about the standards. A small number 
of the SDAs visited were primarily critical, 
perceiving the standards as dominating the 
JTPA system and diverting attention from 
the needs of clients. 

In summarizing service-provider reac- 
tions to operating in the performance-stand- 
ards context, we reflect that most service 
providers do not have a broad view of the 
performance-standards system as a whole but 
rather are familiar with only that version of 
performance expectations that is communi- 
cated to them by the SDA through contract 



goals and requirements. This limited view 
point is sometimes responsible for 
misunderstandings between service providers 
and SDAs that could be avoided if the SDAs 
shared more information about broader 
program objectives with service providers. 

The Standards as a Reflection 
of Program Goals 

Responses to the most general questions 
about the extent of SDA support for the 
standards system fell into four major 
categories: concurrence with the focus of the 
standards (9 SDAs or 30%), mixed or con- 
flicting reviews (7 SDAs or 23%), neutral 
comments (1 1 SDAs or 37%), and openly 
critical reactions (3 SDAs or 10%). 

The nine SDAs in which interview 
respondents expressed general support for 
the standards had two notable features in 
common: they have track records of good-to- 
excellent performance on the standards; and 
with one exception, they actively used the 
standards in planning and managing their 
programs. However, they also differed from 
each other in important ways: urban/rural en- 
vironment, nature of standards-related 
policies in the State, extent of involvement of 
the PIC in policymaking, extent of staff con- 
tinuity ftom CETA, types of contracting and 
service-provider arrangements, and charac- 
teristics of their local economies. 

A vocal minority of these SDAs were 
adamant that the standards are not an impor- 
tant factor in shaping their local program 
goals or operations but are secondary to 
goals established by the PIC or staff. The 
remainder report that they actively use the 
standards to measure the performance of con- 
tractors or consider them a useful barometer 
of the SDA*s overall success. 

One of the most frequent themes among 
those expressmg support for the standards 
was that they enhance the program^s account- 
ability and are useful as a management and 
oversight tool. Some of the comments reflect- 
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ing those views were the following: 

• "You don't know if you win unless 
you keep score." (The quote was 
from a PIC member but represents the 
reactions of SDA staff as well.) 

• They "are a useful measure of what 
we are doing, but I don't know if we 
would do anything differently without 
the standards... number one: we serve 
our population, and number two: we 
do it in a way that meets or beats the 
standards." The SDA that contributed 
this observation has always met all 
seven of the standards, a matter of 
pride to tiiem. 

• "Accountability isn't a dirty word. 
Performance isn't a dirty word.... The 
standards have taken a soft and flabby 
human service program and made it 
into a goal-oriented program that 
works." 

• "In general, the standards are a good 
idea. They are a way of forcing other 
SDAs to run the program as well as 
we aheady do." This is an SDA tiiat, 
at the beginning of the program, 
"designed their services and contract- 
ing around tiie standards" and viewed 
them as a key guidepost for setting 
program priorities. However, their 
current focus is more on managing 
what they view as a good program 
and meeting the needs of their local 
community than on meeting standards 
or earning 6% incentive funds. 

• "They are necessary to ensure the in- 
tegrity of the nPA program." This 
was cited as the unanimous view of 
one SDA's major actors, who also 
cited the performance standards as an 
integral part of local management 
strategy. Respondents in this SDA 
stated that they view good perfor- 
mance — as measured by the stand- 



ards — as a valid indicator that they 
are a success. 

^ They are "useful and valid 

benchmarks which help practitioners 
to be more conscious of outcomes 
and goal achievement. They are not 
crosswise to the goals of the 
program." This SDA also said tiiey 
had never dropped a program they 
thought was good because of the 
standards, and if the issue ever arose, 
they would "sacrifice the standards." 
Their stated goal is o meet, not ex- 
ceed, the standards, and they were 
adamant that the standards "do not 
distort program decisions. We can 
serve those most in need and still 
meet the standards." 
Among the SDA respondents expressing 
overall support for the standards system, a 
second theme emerged, although less fire- 
quentiy than the comments about account- 
ability. This was the view that the perfor- 
mance standards have a positive effect on the 
image of tiie JTPA program, particularly with 
tiie private sector. SDAs made tiie following 
kinds of comments: 

* "The standards get at the heart of 
what JTPA is all about; they have a 
positive effect on people's percep- 
tions of the program; they are a 
barometer for distinguishing us from 
CETA. It isn't possible to reflect 
everyone's concerns in a manageable 
set of standards." This was from an 
SDA that actively uses the standards 
to manage their service-providers' 
performance, setting contract require- 
ments above the standards level. A 
provider comment from that SDA 
was: "the days of serving tiie 
economically disadvantaged as a 
'good cause' are over, because we are 
measured on our accomplishments 
now." 
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• "The greatest influence of the perfor- 
mance standards h^z *^v:en as a public 
relatiois tool for elected officials and 
the private sector." 

• "Especially if you want the private 
sector involved, the program has to 
have a performance orientation and 
measures of progress." The standards 
are useful in selling the program to 
employers and the public, a needed 
focus because JTPA is "still one of 
the best-kept secrets around." 

• "There's no question that perfor- 
mance standards have been a positive 
public relations tool for JTPA, that 
they have helped enhance the 
program's respect in the eyes of 
employers, PICs, and the public at 
large. With the PICs, though, there 
was a learning curve about the stand- 
ards' workings." Other respondents 
worried that the learning curve was a 
big issue: "Our PIC members think it 
is good to have accountability, but 
it's too bad that the rules for im- 
plementing the standards are so 
wacky." 'The PIC sees the standards 
as 'technical stuff." 

A second group of SD As had decidedly 
mixed views about the adequacy of the per- 
formance-standards system. In many such 
SDAs, it was the PIC (or those worlcing most 
closely with them) who expressed the 
strongest positive views of the standards, 
while other staff (sometimes those working 
most closely with service providers) were 
more critical. In some vSDAs, different 
perspectives were expressed by a single 
respondent. On the positive side, many of 
their comments echoed the sentiments ex- 
pressed above. On the negative side, respon- 
dents pomted to »he standards' lack of 
emphasis on quality of training, a disincen- 
tive to serve those most in need, and a tenden- 
cy to resist new or innovative training 



approaches. Some of these mixed views are 
represented by the following comments: 

• "The standards have forced a more 
disciplined approach to the program 
on the part of (the PIC and) SDA 
staff. They are appreciated by the 
business community. However, they 
encourage short-term, quick-fix solu- 
tions and they are blocking needed 
structural changes in our program. 
They are not good indicators of our 
real performance." This SDA 
believes that the emphasis on youth 
expenditures is excessive and is 
taking money from adults who need it 
more. They also want to know how 
the national departure point for the 
youth positive-termination rate "got 
so high," 

• "They address the need of the system 
to be accountable, but they don't tell 
us much about the net impacts of the 
program, and they tilt the balance 
toward meeting employers' training 
needs" (rather than the needs of those 
seeking training). This was from a 
high-performing SDA that stres * 
youth services, has a strong inte 

in "looking good," and believes tnat 
the standards have left them no 
latitude to take risks. 

• "They reflect the goals of JTPA. It's 
great that they exist. All our pro- 
grams should be aimed at meeting the 
performance standards." However, 
another respondent from the same 
SDA stated that "they are not a 
focus — I'm interested in de:iigning a 
program that fits together intellectual- 
ly, not in cooking numbers." This is 
a large and extremely varied program 
with a stated dedication to serving 
anyone who needs service. 

• "The standards get a lot of support as 
a vehicle to encourage cost-effective- 
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ness. They are good for eliminating 
less effective providers. But they dis- 
courage risk-taking and don't permit 
or reward service to those most in 
need." 

"They make the program better-run; 
they prevent the CETA phenomenon 
of ^perpetual trainees.' However, they 
lead an SDA to enroll those with a 
high probability of success, with the 
result that others are not getting the 
services they deserve." The SDA that 
had these comments also added that 
"creaming is inevitable — employers 
do it, why shouldn't we?" Staff differ 
in their opinions about the relative ad- 
vantages (in serving disadvantaged in- 
dividuals) of encouraging them to 
take "any job" and worry about 
upgrading later (with the benefit of 
some work experience) or pushing for 
a higher quality job right away. 

Another SDA expressed support for 
the accountability that the standards 
represent but also stated that they "are 
not all that useful in that they do not 
do a good job of measuring the extent 
to which a program makes a dif- 
ference for the individual trainee. The 
standards are just placement oriented. 
The system is too much focused on 
standards and not enough on people." 
This is an SDA that holds the stand- 
ards responsible for the "virtual shut- 
down of occupational classroom 
training" and the predominance of 
short-term interventions. 

"The standards are good in that they 
give you the numbers you need to 
keep track of the program, but they 
don't give you a real feel for the 
quality of the program or the jobs 
people are getting. Even performance 
on the wage standards doesn't tell 



you about the fringe benefits people 
are getting." 

• Another SDA characterized the stand- 
ards as salient to their goals but in- 
complete. They cited the wage 
standard as the only one that high- 
lights quality of services and out- 
comes. They do not use the standards 
as a planning tool or as a basis for set- 
ting program objectives but rather as 
a mechanism for quality control after 
the fact. In fact, they consciously try 
not to let the performance standards 
drive their program. They hope the 
standards will let them operate "busi- 
ness as usual" and still get some in- 
centive funding. However, another 
comment from the same SDA was 
that the standards "can't help influenc- 
ing service design, because they have 
gotten tighter and tighter every year. 
We have had to delete the most expen- 
sive training options." 

The mixed overall view of the standards 
also translated into mixed views about how 
they affe'.:ted the public perception of the 
program. As far as some SDAs were con- 
cerned, the standards were a mixed blessing: 

• "We are trying to change our image 
to be a cost-effective program in- 
volved with the private sector, and 
the performance standards are very 
helpful for that purpose. But they are 
too neutral with respect to who is 
served; we don't feel rewarded for 
taking the initiative to focus attention 
on hard to serve clients." 

A third group of SDAs had fewer general 
conmients about the appropriateness of the 
standards. They can be characterized as simp- 
ly accepting the standards system as given. 
Their comments were concentrated instead 
on discussions of how they had learned to 
adapt to the system. This was a varied group 
of SDAs, with different levels of PIC in- 
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fluence and CETA continuity and different 
local environments. 

Of the 10 SDAs in that category, 7 de-em- 
phasized the importance of performance 
standards in their decision making. However, 
some of those same SDAs admitted that earn- 
ing the maximum amount of 6% funding was 
an important goal in a world of shrinking al- 
locations. (One of these SDAs even gives 
staff raises that depend on the level of 6% 
award achieved.) One of the SDAs in this 
category explained this seeming contradic- 
tion by stating "we go about business as 
usual and then hold our breath." Those SDAs 
also cited a strong role of the PIC or LEO in 
setting program priorities. 

The other three SDAs cited a stronger in- 
fluence of the standards in shaping their local 
programs. One said that "they drive our 
whole program, and influence our choices 
about types of training, emphasizing the 
shortest, cheapest programs that art still 
aimed at high wage jobs." Another said that, 
however important it is to maximize 
measured performance, that goal does not 
keep them from serving high-risk clients nor 
induce them to "do job club or money 
wasters." The single feature that these three 
SDAs have in common is that they have all 
experienced some difficulty meeting one or 
more standards. 

The final group (3 SDAs) were primarily 
critical in their comments about the perfor- 
mance standai'ds system as a whole. The 
reasons they gave were the following: 

• "The performance standards intensify 
system inertia — where there is a bias 
against change, the standards exacer- 
bate it and are used to justify a reac- 
tive posture." (This comment was 
from a respondent in an SD A judged 
to have adopted a reactive posture to 
operating their program). 

• "We are concerned that the standards 
focus more on quantity than quality; 



they do not reflect the challenge of 
serving those most in need; they 
provide incentives for numbers-play- 
ing; they discourage attention to up- 
ward mobility; they promote 
quick-fix solutions to employment 
needs." This SDA is serving an in- 
creasingly needy population because 
of the decline in unemployment, and 
respondents report that they arc nerv- 
ous about the effect of this change on 
their measured performance. 

• "The standards get too much atten- 
tion in the system. It's inevitable. Our 
survival depends on meeting thenL 
They drive the system too much." 
This SDA gives staff a $500 bonus if 
performance standards are met. One 
respondent in the SDA believes that 
the system is overreacting to the per- 
ception of CETA as "sloppy" by run- 
ning a program that is afraid of 
innovation and places too much em- 
phasis on "running a tight ship." 
In other words, case study interviewers 
encountered nearly as many different overall 
reactions to the performance standards as 
there were respondents with a genuine inter- 
est in program management and performance 
issues. The views expressed above were 
directed at the performance-standaixis system 
as a whole. In addition, SDAs were asked 
their opinions about specific standards and 
the ways in which specific standards affect 
local operations. 

The stand^ds that occupy the greatest 
share of management attention at the SDA 
level fit into three categories: (1) those that 
the SDA perceives as being most difficult to 
meet— most frequently wage and youth 
standards, (2) those that the State has man- 
dated or emphasized in the incentive for- 
mula—usually the adult standards, and (3) 
those that local decision makers view as im- 
portant — predominantly the entered-employ- 
ment and cost standards. 
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Several respondents stated that the stand- 
ards are, or ought to be, equally important. 
As firequently as not, they went on to qualify 
such statements by pointing out those stand- 
ards that have been tracked with most care at 
the local level: Examples cited by particular 
SDAs were wage (hardest to meet) and cost 
(most heavily rewarded in the incentive for- 
mula). Cost also emerged as one of the stand- 
ards emphasized by those local PICs that are 
anxious to spread a limited amount of money 
to serve as many persons as possible. Some 
such PICs appear to be rethinking the latter 
emphasis, given increased Federal emphasis 
on quality of trainings Others regret the em- 
phasis placed on cost. 

The youth standards were cited as a par- 
ticular problem for some SDAs: those that 
were late in developing YECs or that were 
still resisting developing them, and one SDA 
in which staff believe that the needs of disad- 
vantaged adults are more urgent than those of 
youth. (On the other hand, one SDA respon- 
dent stated that "we believe that money spent 
on youth is well spent," and thus that the 
youth standards are important as measures of 
success.) 

Average wage was singled out as relative- 
ly important in many SDAs, although for dif- 
ferent reasons. Some said that achieving high 
wages is a matter of local policy priority. 
Others describe average wage as the target of 
considerable attention because of being dif- 
ficult to meet. Pressiire to meet wage expecta- 
tions was also cited by service providers as 
one factor influencing them to select relative- 
ly job-ready clients. Some SDAs were able to 
improve their performance on the wage stand- 
ard by relying on OJTs in which they were 
able to negotiate wage increases during the 
period of training. One SDA offered the 
opinion that an overemphasis on high wages 
may have cost them some placements. Final- 
ly, one respondent stated that, although the 
SDA had complained about the wage stand- 
ard, they are glad they were "pushed" to 



achieve higher wage outcomes for their par- 
ticipants. 

Levels at Which Standards Are Set 

Case study respondents were also asked 
about the levels at which the standards are 
set. SDA reactions to those levels followed 
patterns that appear to be shaped largely by 
their history of performance, both overall and 
with respect to particular standards. The 
other factor that appears to have influenced 
SDAs is differences in State pertbrmance- 
standards policies. SDAs in those States that 
emphasize overperformance in their 6% in- 
centive formulas were more likely to express 
concerns about high or escalating standards 
levels. 

About half of the case study SDAs 
showed a lack of concern for the numerical 
levels at which their standards are set but ac- 
cepted them as a matter of course. For the 
most part, these were SDAs that had ex- 
perienced little or no difficulty attaining the 
expected levels of performance. One respon- 
dent in such an SDA went so far as to State 
that they like the fact that the standards are 
easy to meet, but they are "not stupid enough 
to say the standards are too low." Another 
stated that, in general, the standards are mean- 
ingless if everyone meets them. On the other 
hand, if they are set too high, SDAs will have 
an increased incentive to cream. Both of 
these SDAs are in a State that has placed 
very litde emphasis on performance. 

Respondents in nearly one-fourth of the 
sample SDAs commented on the increases 
over time in the levels of expected perfor- 
mance. All of these SDAs are in States in 
which part or all of the 6% incentive funding 
is awarded based on competition among 
SDAs. One such SDA characterized the ef- 
fort to meet the standards from year to year 
as "trying to hit a moving target." At least 
four others made specific reference to the 
"ratcheting" effect that results from using a 
national departure point based on the pre- 
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vious years' performance. One respondent 
went on to say that the overall increases in 
the levels at which standards are set are 
heightening the emphasis on quick-fix solu- 
tions, an emphasis at odds with the remedia- 
tion focus of the program. The increasing 
level of the standards, in that respondent's 
opinion, creates too much of a tendency to 
make decisions based on "the quicker the bet- 
ter" criteria, to cream within target groups. 

The remaining SD A reactions consisted 
of complaints that a specific standaid is too 
high, difficult for them to meet, and therefore 
absorbing a great deal of management atten- 
tion. This group included all four of the 
SDAs in the sample State that most strongly 
stresses overperformance. One respondent 
noted that, if one standard is too difficult to 
meet, it will skew the program. 

As stated above, the most frequently men- 
tioned "problem" standard was wage, cited 
by at least seven case study SDAs as being 
the most difficult to meet. The youth positive 
termination and cost per positive termination 
were also fi:^uently cited. Some SDAs said 
that these standards make it difficult to serve 
high-risk youth. One went further to say that 
even easing the pressure on the youth cost 
standard — while keeping the positive ter- 
mination rate high — won't do any good: 
"throwing extra money at them (hard-to- 
serve youth) won't solve the problems of the 
20% who don't want to be helped." 

In general, then, case study SDAs were 
equally divided between those in which the 
level of the standards is accepted without 
question and is not problematic and those in 
which either some or all of the standards are 
seen as too high or rising too quickly. The lat- 
ter group in turn was divided about equally 
between SDAs concerned about the escala- 
tion over time in the levels of all standards 
and those concerned more with possible dis- 
tortions that occur when a particular standard 
(or standards) is harder to meet than others 
and, therefore, from the SDA's point of 



view, too high. Those issues were focused on 
a variety of specific standards, most frequent- 
ly the wage and the youth standards. 

Reactions to DOL 
Adjustment Model 

The most notable feature of SD A com- 
ments about the Secretary's model was their 
diversity. A few respondents' reactions were 
echoed in more than one SDA, but most 
were idiosyncratic and reflected the par- 
ticular challenges faced by the SDAs in our 
case study sample. The most frequent 
response was "we wish the model took our 
own situation into account with more 
precision." Some of the sentiments that were 
repeated in more than one SDA were the fol- 
lowing: 

• The adjustment model is a "reasonab- 
ly sensible thing to be doing." Two 
SDAs expressed appreciation for the 
efforts made at the Federal level to 
set up a system to adjust for local cir- 
cumstances. 

• Several rural SDAs noted what they 
believed to be a bias in the model 
toward urban areas, with not enough 
adjustment for the difficulties en- 
countered in a spread-out SDA. Par- 
ticular targets of such comments were 
the wage and cost standards, which 
they believe are particularly difficult 
to meet in rural areas. 

• Other comments about the adjustment 
for area wages were frequent: 

- Average wage used to be a 
problem for one SDA, because of 
the high manufacturing wages in 
the area, but that has in^roved 
since the model factors have 
changed. 

- The model is not adequately adjust- 
ing for differences in the cost of 
living. 
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- The model does not adjust ade- 
quately for the difference between 
average wage and entry-level wage, 
but whatever the SDA loses in the 
wage standard, it gains in the cost 
standard. 

Several comments related to adjust- 
ments for the population served: 

- The model does not really reflect 
regional differences in populations. 
For instance, the difficulty of serv- 
ing Hispanics in some areas is 
much greater than others. 

- Two SDAs questioned State-level 
changes in the model for serving 
minorities: "at least the national 
model has statistics behind it." 

- The model does not adequately ad- 
just or reward service to the hardest 
to serve. 

- In an area with few AFDC 
recipients, the "small numbers 
problem" makes the level of some 
of the standards quite volatile. 

Some comments were shaped by the 
service mix in particular SDAs: 

- There should be an adjustment for 
SDAs that conduct adult remedia- 
tion programs. "WeVe not afraid to 
serve the worse-off, just afraid the 
model will (or won*t?) catch up 
with us." 

- One of our case study States has 
developed and is testing its own 
model, one that some SDAs believe 
more closely reflects local cir- 
cumstances in their State. That 
model includes some adjustment 
for the types of program activity 
used. 

Finally, there were a few miscel- 
laneous reactions to the adjustment 



model and pending changes in the 
model: 

- Measuring reading level was seen 
by some SDAs as a problem, partly 
because they believe that it is a 
poor predictor of success and partly 
because of the cost of collecting the 
data. 

- The differences the model makes 
(in the standards level) are "too 
small to bother with." 

- The commuter ratio in the model 
for average wage acts against one 
SDA that includes out-commuters 
but which has a local policy to 
provide workers for local area 
employers. That SDA will re-ex- 
amine its policy toward local 
employers. 

- There is no way to factor in the 
fringe benefits received, which is a 
strong PIC priority in one SDA. 

- One respondent noted that extreme 
values in the model made it pos- 
sible for another SDA in the State 
to "make out like a bandit" in the 
competition for 6% funds. 

- The new terminee/participant ratio 
will cost one SDA $500 on the cost 
standard because its practice is to 
terminate all participants by the end 
of the program year. 

- One SDA concurs with three of the 
new factors: long-term welfare 
recipient, never employed, and 
reading level. However, they report 
that contractors resist the reading- 
level measure, apparently out of 
fear that they will have to educate 
all participants (whether they want 
education or not) or that they will 
have a disincentive to enroll those 
with deficient reading skills — the 



... 183 223 



Qualitative Analysis 



provider would have been unaware 
of the deficiency except for the 
new data gathering requirement 
As part of our survey of all SDAs, direc- 
tors were asked to rate how well each DOL 
model adjusted for the SDAs' circumstances. 
Table 35 presents these results. Overall, 
more than 50% of the respondents rated the 
models' adjustments as either good or excel- 
lent. Directors generally rated adjustments to 
the adult standards higher than the youth 
measures. The adjustment model for tlie 
adult wage at placement, however, was rated 
poor by 18% of the directors * 

Technical Assistance Needs 

Only five SDAs expressed an interest in 
receiving technical assistance on the model. 
For example, one SD A would be interested 
in setting up the model on their computer sys- 
tem to work through some of the alternatives 
in detail.** Another SDA would like to un- 
derstand why factors and weights change 
fix)m year to year, citing the difficulty of 
planning without knowing what the factors 
and weights will be. Finally, one central-city 
SDA wanted to know the logic bdiind some 
of the specific factors in the model. Why, for 
instance, does increased population density 
raise the expected placement rate? This direc- 
tor hypothesized that the relationship be- 
tween density and ease of placement was 
bimodal rather than linear, with very rural 
and very urban places having problems, be- 
cause "all the jobs are in the suburbs now." 

In addition to the SDAs where technical 
assistance was specifically mentioned, quite 



a few other respondents indicated some 
misunderstanding about how die model 
works. For example, one SDA feared that, if 
tiiey improved their measured performance, 
their standards would become higher in sub- 
sequent years. Another only recentiy under- 
stood die rationale for recalculating the 
model at the end of the program year. 

In our survey of all SDAs, directors were 
asked questions about the meaning of the ad- 
justment weight in the models. Approximate- 
ly 75% of the respondents understood that 
the negative weight for welfare recipients in 
the entered-employment rate model indicated 
welfare recipients were less likely to enter 
employment and diat die more welfare 
recipients die SDA served, die lower dieir 
standard would be. Approximately one-third 
responded, however, that the adjustment also 
meant their standard would be easier to meet 

Thus, a substantial majority of SDAs un- 
derstand the adjustment model, although 
there is still considerable misunderstanding. 
In our quantitative analysis, however, we 
found that the SDA's score on this set of 
questions was not related to the types of 
clients served. 

Data Definition Concerns 

Aside from the concerns expressed over 
die ability of the Secretary's model to adjust 
for SDA circumstances, a number of respon- 
dents had concerns about specific data items 
used in die model and JASR. "Handicapped" 
was the data item most often mentioned, and 
it is not clear whether all SDAs are fully cap- 
turing die number of handicapped clients 



♦It is interesting to note that the adjustment model explained over 62% of the variation in average wage rates 
across SDAs in PY 86, higher than any other model. The directors' greaUr : dissatisfaction with the adjustments 
probably reflects the fact that SDAs have less control over the wage-rate outcome than the other perfoimance 
measures. 

**This SDA had enrolled some Tide n-B in-school youth into Tifle II-A programs. This action q)parenfly 
made YPTR more difficult for them to meet (or so they thought). sinf;e this group of participants included fewer 
minorities, dropouts, and AFDC recipients than their other Tifle II-A jiarticipants, and tiius flie level of tiie adjusted 
standard was raised. Had they been able to project in advance tiie effe<;ts of tiiis change in participant characteris- 
tics, tiiey might not have made the enrollment change. 
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Directors' Ratings of How Well Each Model Adjusts 
for the SDAs' Circumstances 



Excellent Good Fair Poor 



Adult Models 



Entercd-employmait rate 


8.3% 


56.2% 


28.0% 


6.2% 


Average wage at placement 


6.4 


43.3 


30.4 


18.0 


Cost per entered employment 


10.1 


53.1 


24.7 


10.1 


Welfare entered-employmentrate 


6.7 


48.3 


32.6 


10.5 


4th Models 










Entered-employment rate 


5.6 


47.7 


31.0 


13.1 


Positive-termination rate 


4.6 


46.3 


31.4 


15.4 


Cost per positive termination 


7.2 


47.8 


28.0 


14.4 
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they serve; a few providers reported that 
clients felt stigmatized if categorized as hand- 
icapped and would not sign their intake 
forms if that box was checked. Another 
client characteristic that caused some con- 
cem was "offender." One State visited had es- 
tablished a task force to look at definitions, 
which decided to use the Federal definitions 
but to issue clarifications to their SDAs. 

A few respondents worried about the lack 
of outcome definitions. One director was con- 
cerned that "just going to work is a place- 
ment," and wanted some more precise 
definition. Others were troubled by the lack 
of uniformity in YECs across SDAs; as one 
director said, "we aren't all playing by the 
same rules." 

Aside from these specific concerns, most 
respondents had no concerns about the con- 
sistency of the data presently used in JTPA 
programs. The data required for the new fol- 
low-up standards had little visibility in most 
SDAs visited, because nearly all of the case 
study States collected that information in a 
cttitralized way. However, two SDAs in the 
one State that required SDAs to do their own 
follow-up were concerned that welfare 
recipients would not reliably report their earn- 
ings and employment at follow-up, for fear 
that their benefits would be jeopardized. 

Reporting Concerns 

For the most part, respondents did not 
feel overburdened by JTPA reporting require- 
ments, at either the SDA or the service- 
provider level. When asked about this aspect 
of JTPA programs, most service-providers 
took the attitude that the reports usually re- 
quired of them — attendance forms, progress 
reports, placement outcomes — were 
reasonable given the sort of program they 
were participating in. Only one service 
provider complained that documenting pay- 
ment points was a burden, and one in-school 
youth provider wondered if paperwork re- 
quirements prevented more principals from 



participating. Two service providers men- 
tioned that reporting was a burden when 
clients were receiving services through two 
funding streams that had different forms re- 
quiring the same information. 

SDAs usually had automated systems 
that allowed them to quickly and easily trans- 
mit reports to their States and, therefore, did 
not feel that reporting was an inconvenience. 
In general, these MIS systems had been 
designed by the States, and SDAs were satis- 
fied with their overall operation, feeling that 
they got useful and timely summary informa- 
tion back from the MIS as well as having few 
worries about submitting the raw data. 
However, one respondent strongly disagreed 
with overall lack of concern about the report- 
ing burden: he had calculated that JTPA 
programs required four times as many 
reports as CETA and believed that most 
SDA directors merely passed on requests for 
more data to their subordinates vdthout think- 
ing about the burden involved. 

Anticipated Effects 
of New Standards 

Discussions of the effects of the post- 
program standards to be implemented in PY 
88 took a different turn depending on 
whether the discussion was focused on the 
general effects of the standards on the JTPA 
program as a whole or the specific an- 
ticipated effects on the SDA's own program 
and level of measured success. A large 
majority of the respondents who addressed 
systemwide effects said they supported the 
adult post-program standards. Even those 
who admitted to some uneasiness about im- 
plementation added that in principle they sup- 
port the new standards. Some of the overall 
views they expressed were the following: 

• They will show the world the value 
of real training; they will enhance the 
reputation of the JTPA program. 

• Our response is enthusiastic: post- 
program standards will be a better 
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measure of whether the training 
"sticks"; they are consistent with this 
SDA's goals; in fact. DOL should 
mandate that they be used. 

• They are viewed as "healthy"; if any- 
thing, it would be good to use a 
longer time period after training; 
other useful measures would be a wel- 
fare dependency reduction and the ad- 
ditional number of participants an 
SDA was able to serve by linking 
with other sources of funding for sup- 
portive services. 

• We agree with the focus on follow- 
up; we already track post-program 
outcomes for our participants. 
(Several SDAs expressed variants of 
this sentiment, a few adding that they 
use a 2-week or 30-day post-place- 
ment retention point as a basis for 
payment in fixed-unit-price contracts.) 

• They will be a more accurate reflec- 
tion of the quality of the program (at 
least three SDAs). 

• We support measuring follow-up out- 
comes. We are aheady reducing our 
job-search assistance program be- 
cause we have noticed the low reten- 
tion record of those who participate in 
that program only. 

Some SDAs responded more in terms of 
the expected practical effects of follow-up 
standards on their own SDA's operation than 
in terms of their reactions to follow-up stand- 
ards in principle. The majority of these 
responses were defensive in nature. One 
respondent characterized the follow-up 
employment standard as dangerous and not a 
good management tool, because the SDA has 
no ability to control post-program outcomes. 
Follow-up earnings measures were more easi- 
ly accepted, because they reflect the ex- 
perience of employed clients. The follow-up 
standards were dubbed "meaningless and in- 
effective'' by an SDA that acknowledged 



being "uneasy" about their practical effects. 

The follow-up standards, another SDA 
noted, will create conflicts with current pres- 
sures to serve more participants at low cost 
and with the severe barriers to employment 
presented by the client population (which this 
SDA does not feel it has enough time to 
remove). Post-program outcomes are likely 
to cause problems for this SDA, although 
local priorities are already shifting somewhat 
toward more intensive services that will help 
them meet these standards. 

By contrast, one SDA welcomed the chal- 
lenge of implementing the new standards, 
believing that they will make the SDA serve 
fewer people with more intensive and more 
expensive services. This was a direction they 
are already moving in response to State 
policies emphasizing quality and de-empha- 
sizing cost. 

At t?ie time of our case study visits, many 
States had not yet decided how many or 
which of the new standards they would use. 
Thus, some of the case study SDAs were con- 
cerned or uncertain about how their States 
would choose to implement the new stand- 
ards. Others worried about procedures for 
data collection. One SDA expressed concern 
about who will collect follow-up data: if ser- 
vice providers are required to do it, there may 
be incentives for them not to be objective in 
reporting findings. If the SDA or State does 
follow-up, they will have more difficulty 
tracking and finding lerminees and getting 
them to respond meaningfully. Several ser- 
vice providers who served developmentally 
disabled or mentally ill individuals worried 
that centralized follow-up would be inap- 
propriate for their clients. 

From those SDAs in which respondents 
singled out the new youth standard and chan- 
ges in competency requirements, the reac- 
tions were most frequentiy negative. One 
SDA, however, expressed support with the 
general thrust of the changes. This SDA ac- 
knowledged that it had initially developed 
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YECs only because of the need to meet the 
positive-tennination-rate standard, but staff 
now believe that they are appropriate and 
good. They support stronger definitions, 
believing that some areas are abusing the 
YECs, However, the requirement to achieve 
two of the three competencies will be dif- 
ficult for those programs in which occupa- 
tional skills training is on an individual basis. 

Another SDA was worried about the re- 
quirement to document achievement of two 
competencies. It is not fair, they said, chang- 
ing definitions and standards in the same 
year. This does not allow SDA? to gather 
baseline information for planning purposes. 

Two other SD As were even more 
adamant in their opposition. "We strongly op- 
pose the new youth requirements. They 
(other SDAs) will just manipulate YECs to 
meet them. YECs were hard to set up, and 
now there^s an incentive to change them." 
The other SDA characterized the new youth 
employability enhancement standard as "a 
monster, a creation of the education people.'' 
Staff in this SDA believe that the employ- 
ment focus of the program should be em- 
phasized: "what counts are Jobs.'* 

In general, SDA reactions to the new 
standards to be implemented in PY 88 fell 
into two categories: (1) generally positive 
reactions to the post-program standards for 
adults in principle , with some SDAs express- 
ing doubts about what their practical effeins 
might be, and (2) a lack of enthusiasm for 
changes in the youth standards. 

Service-Provider Opinions of the 
Performance-Standards System 

Interviews with case study service 
providers indicated that as a group they had a 
limited view of the performance-standards 
system under which the SDAs operated. Ser- 
vice providers were by and large unaware of 
the existence of performance standards in 
general, although a few knew that the SDAs 
were somehow judged on outcomes and 



received bonus funds based on those out- 
comes. Even when service-provider staff 
knew of the existence cf a performance- 
standards system, they were usually unaware 
of the levels at which they were set. An ex- 
ception to this overall pattern existed in a 
few SDAs where service providers served on 
PICs. However, for the most part, the service 
providers tended to know only the terms of 
their own contracts. 

Because of the narrow perspective of 
most program operators, they often had a 
hard time understanding why SDAs took cer- 
tain actions to achieve overall management 
goals, including performance goals. For in- 
stance, service-provider staff were often 
resentful when SDA intake and assessment 
staff referred participants that the service 
provider had recruited to other programs or 
activities within the SDA rather than back to 
the service-provider^s program. (SDAs some- 
times diverted the most job-ready clients 
referred by providers into a job-search com- 
ponent run by the SDA.) Service providers 
usually were unaware of the range of ser- 
vices being offered by the SDA and, there- 
fore, did not realize that a more appropriate 
service might exist. However, their distress 
was understandable in SDAs where un- 
employment was low and recruitment a 
problem, because service providers in these 
circumstances were desperate for clients. 

Sometimes service providers were 
greater risk takers than SDAs when it came 
to recruiting clients, because of their conunit- 
ment to serve the hard to serve. One service 
provider was angry when potential par- 
ticipants were rejected by SDA intake staff 
on the basis of purely subjective judgments 
about their likelihood of program comple- 
tion; the provider reasoned that the perfor- 
mance-based contract gave them the right to 
take their own risks, especially when there 
were training slots going unfilled. 

Another procedure that helped SDAs 
meet their own performance goals but caused 
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resentment among providers was the com- 
mon practice of SDAs taking credit for place- 
ments made by providers ihat were not 
eligible for reimbursement under r!ie terms of 
the provider's contract. Usually tne^e were 
placements that were non-training related, 
part time, or at a wage below that specified in 
the contract. One service provider negotiated 
partial payment for part-time placements 
after learning that the SDA could take credit 
for those placements; they reasoned that 
many employers wanted part-time employers 
and many participants wanted those jobs. 

Because service providers did not always 
know the performance requirements under 
which the SDAs operated, they did not under- 
stand why some of their contract terms were 
important to the SDAs. For instance, a few 
service providers that were not paid for place- 
ments below a certain wage rate thought that 
many potential clients could only be ex- 
pected to obtain minimum wage jobs and dis- 
liked having to turn them away due to the 
wage requirements in their contracts. 

Although many service providers did not 
fully understand the performance-standards 
system, they nevertheless appreciated the 
chance to be judged on outcomes, either by 
operating under performance-based contracts 
or through careful performance reviews. All 
kinds of organizations reflected the attitude 
that the JTPA system was different from the 
CETA program because outcomes were im- 
portant. One private nonprofit provider said, 
"JTPA is a good program because it takes the 
money and gives it to the private sector, 
which knows how to do things. This is like a 
business; if we can't perform, then we 
shouldn't be in the business of training 
people." A CBO with a history of service that 
went back to CETA liked performance-based 
contracts "because they make you feel like 
you're running a business, because you 
provide a service and get paid for it." 
However, several service providers reported 
that they would not undertake new or innova- 



tive programs on a performance-based con- 
tract, only programs with a proven track 
record. 

As discussed above, performance-based 
contracts were not new in many SDAs. What 
was new very often was an increased em- 
phasis on lower costs, as SDAs felt driven by 
their declining allocations to spread resources 
thinner and thinner. Service providers felt 
this pressure very strongly, and many wor- 
ried how much more they -^ould cut. 

When performance-based contracts were 
new, their implementation seemed to proceed 
more smoothly the more the SDA staff ex- 
plained the reasons for their desired contract 
terms. Several educational institutions strong- 
ly resented the placement holdbacks that 
were imposed on them without explanation. 
These respondents either felt that what the 
SDAs were doing was illegal or that 
holdbacks constituted arbitrary reductions in 
tuition that was rightfully theirs. However, 
many other educational institutions hao ad- 
justed to the notion of placement holdbacks 
without problems; one program operated by a 
community college was proud of the fact that 
they were more accountable for taxpayers' 
money than, say, the English department. 

Service providers that were unhappy with 
performance-based contracts were usuaJly in 
SDAs with low unemployment. Recruitment 
was difficult in these SDAs, which often 
meant that service providers would not have 
enough clients to obtain tiie full amount of 
their contracts. These providers felt that their 
unit costs had been negotiated so far 
downward that tiiey were operating "on the 
edge" anyway, so the prospect of not having 
enough clients to fill slots was especially 
disheartening. In addition, the improved 
economy generally meant that tiiey had to 
serve increasingly more difficult clients at in- 
creasingly lower costs, a situation that left 
them feeling squeezed and out of control. 
The same CBO that was glad to be operating 
under performance-based contracts in order 
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to be judged on outcomes also pointed out 
the down side: "Unit pricing has advantages 
when you have people to enroll, but if you 
don't, you might take a loss. Since we don't 
have clients, Tve given my instructor 
notice — if we were on cost-reimbursement 
maybe I could use her to do recruitment." 

Although service providers shared with 
SDAs the goal of achieving good outcomes, 
they often felt that their very survival 
depended on being paid the full amount of 
their contract. This differing perspective ex- 
plains why some providers were thinking 
about leaving the JTPA system — the flow of 
funds was simply too uncertain. When ser- 
vice providers felt assured of their own sur- 
vival, however, they were willing to go 
beyond the terms of their contracts. We saw 
no evidence, for example, that any service 
provider stopped placing clients when the 
maximum amount of their contract was 
reached. In fact, many were proud to have 
achieved higher placement rates than were 
called for in their contracts. 

Summary and Conclusions 

About one-third of the SDAs visited ex- 
pressed general support for the focus on ac- 
countability that the performance standards 
reinforce; these SDAs also supported their 
usefulness in enhancing the credibility of the 
JTPA program. 

About one-fourth of the SDAs visited 
gave the performance standards mixed or 
conflicting reviews: while they appreciated 
the increased emphasis on outcomes, they 
also pointed to the standards' lack of em- 
phasis on quality of training and disincen- 
tives to serve those most in need or to try 
innovative training approaches. 

About one-third of the SDAs were 
generally neutral about the standards and had 
mostly concerned themselves with adapting 
to them. 



A final group of three SDAs were 
primarily critical in their comments about the 
performance standards system as a whole, be- 
cause they perceived them as dominating the 
JTPA system and diverting attention from 
the needs of clients. 

SDAs were about equally divided be- 
tween those in which the level of the stand- 
ards is accepted without question and is not 
problematic and those in which either some 
or all of the standards are seen as too high or 
rising too quickly. The latter group in turn 
was divided about equally between SDAs 
concerned about the escalation over time in 
the levels of all standards and those con- 
cerned more with possible distortions that 
occur when a particular standard (or stand- 
ards) is harder to meet than others and, there- 
fore, from the SDA's point of view, too high. 
Those issues were focused on a variety of 
specific standards, most frequently the wage 
and the youth standards. 

SDAs had diverse comments about the 
DOL adjustment model, but no overall pat- 
tern emerged. SDAs were often concerned 
about a single factor or standard for which 
the model did not seem to fully take their cir- 
cumstances into account. The model is still 
mysterious to some, and more technical assis- 
tance was requested by a number of SDAs; a 
few were operating with obvious 
misunderstandings of the model. 

For the most part, respondents did not 
feel overburdened by JTPA reporting require- 
ments, at either the SDA or the service- 
provider level. SDAs generally had 
State-designed information systems that al- 
lowed them to both input and retrieve the 
data they wanted without trouble. Service 
providers felt that the SDAs made reasonable 
data requests. However, several SDAs ex- 
pressed concern about the extent of the 
documentation required of clients for 
eligibility purposes. 
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SDAs expressed a mixture of support for 
the intent of the new post-program measures, 
skepticism about how they will work in prac- 
tice, and difficulties with the new youth 
standards and requirements. 

Most service providers do not have a 
broad view of the performance- standards sys- 
tem as a whole. Rather, they are only familiar 
with performance expectations that are com- 
municated to them by the SDA through con- 
tract goals and requirements. This limited 
view point is sometimes responsible for 
misunderstandings between service providers 
and SDAs that could probably be reduced if 
the SDAs shared more information about 
broader program objectives with service 
providers. 



This chapter has indicated that SDAs and 
service providers had a variety of reactions to 
and opinions about the performance stand- 
ards. It is important to remember, however, 
that performance standards are only one 
aspect of the JTPA system. When the case 
study interviews turned fix)m an overt discus- 
sion of the standards and their effects to a 
more detailed examination of the SDAs' and 
service providers* actual service program and 
client population, the influence of the stand- 
ards often receded or were overwhelmed by 
other factors, such as the local environment, 
PIC and State client goals and priorities, and 
other Federal requirer lents such as the neces- 
sity to spend 40% of the Title II-A allocation 
on youth. 



191 231 



Section D. 

Summary of Study Findings and Conclusions 



2S2 

ERIC 



XIIL Conclusions about the Impact of 
Performance Standards on Clients, 
Services, and Costs 



Results from the 
Quantitative Analysis 

The quantitative analysis identified how 
variations in State performance-standards 
policies have affected the clients, services, 
and costs of local JTPA programs. The quan- 
titative analysis traced the effects of State per- 
formance-standards policies in two ways. 
First, we measured the total effects of State 
policies on clients, services, and costs, con- 
trolling for a variety of other variables, in- 
cluding local economic conditions, 
characteristics of the JTPA-eligible popula- 
tion in each local area, and variations in the 
roJ^s played by PIC members and LEOs. 
Se::c>nd, we examined the impact of several 
SDA design decisions on the types of clients 
served, including procedures to recruit and 
enroll different type of clients, criteria for 
selecting clients, the type of service providers 
used, and the type and terms of contracts. We 



then examined the influence of performance 
standards policies on those design decisions. 

The analysis of the impact of perfor- 
mance-standards policies showed that several 
dimensions of State performance-standards 
policies do significantly affect the extent to 
which SDAs serve members of hard-to-serve 
groups and the types and average duration of 
JTPA services provided in the local area. 
Policies that increase service to hard-to-serve 
clients in JTPA include the following: 

• Use of the POL adjustment model 
significantly increases the percentage 
of both adult welfare recipients and 
dropouts served. Use of the adjust- 
ment model also significantly in- 
creases enrollment of youth with 
"other barriers to employment" (that 
is, offenders, handicapped in- 
dividuals, and limited-English 
speakers). 
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• State policies on adjustments beyond 
the model have less consistent effects 
but increase services to adult minor- 
ities and increase services to youth 
welfare recipients and minorities. 

• State policies on use of 6% for hard 
to serve (setting aside 6% funds for 
services to hard-to-serve groups, re- 
quiring that some 6% funds be used 
to serve hard-to-serve groups, or ex- 
empting 6%-funded projects from the 
performance standards if they are 
used for hard-to-serve groups) in- 
crease service to adult welfare 
recipients and to adults with "other 
employment barriers" but decrease 
services to women. For youth, these 
policies increase enrollment of wel- 
fare recipients and in-school youth. 

• State policies encouraging services to 
specific client groups (a formal policy 
statement or a 6% policy that rewards 
service to a particular group) are ef- 
fective in increasing enrollment of 
welfare recipients, dropouts, and 
older adults. 

Policies that decrease service to hard-to- 
serve clients in JTPA include the following: 

• Incentive policies that emphasize ex- 
ceeding standards include (1) reserv- 
ing most of the incentive awards for 
substantially exceeding standards, (2) 
having SDAs compete for 6% funds 
by comparing performance levels 
across SDAs, (3) not "capping" the 
performance level beyond which addi- 
tional incentives would not be earned, 
and (4) requiring a large number of 
standards be exceeded to qualify for 
incentive awards. These policies sig- 
nificantly reduce service to welfare 
recipients and minorities for both 
adults and youth and reduce service 
to older adults and in-school youth. 

• Placing greater weight on the cost 



standards reduces service to adult wel- 
fare recipients and adult dropouts. 
For youth, this practice was estimated 
to reduce services to welfare 
recipients, dropouts, anc youth with 
other barriers to employment. 
These results indicate that some State 
policies have their intended effects but that 
the incentive policies in some States are unin- 
tentionally reducing service to some hard-to- 
serve groups. 

State performance-standards policies also 
influence the types of services offered in 
JTPA programs, including the relative em- 
phasis on types of program activities and the 
average duratio*i of JTPA services. 

Several State policies are generally as- 
sociated with more intensive or longer term 
services: 

• Adjustment procedures . Use of the 
DOL model significantly increases 
the length of adult programs and 
reduces the amount of job-search as- 
sistance provided to youth. State 
policies thai specify procedures for 
additional adjustments significantly 
increase provision of basic skills train- 
ing to youth and reduce job-search as- 
sistance, although they have an 
unexpected negative impact on the 
length of participation. 

• State policies for hard-to-serve 
groups . State policies for serving wel- 
fare recipients significantly increase 
the amount of basic skills trainins of- 
fered for both adults and youth and 
reduce the amount of OJT, sig- 
nificandy so for adults. Policies for 
serving dropouts significantly in- 
crease the average length of services 
for adults and youth, but policies for 
serving dropouts also increase the 
provision of job-search assistance, 
reducing OJT for adults and work ex- 
perience for youth. 
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• Placing greater weight on the wage 
standard Policies emphasizing the 
wage standard significantly increase 
provision of classroom training in oc- 
cupational skills and reduce OJT for 
adults. 

State policies that were found to reduce 
sei * to hard-to-serve groups also affected 
program services. These policies generally 
either reduced the intensity of services or in- 
creased the employment focus of the 
program. These policies include the follow- 
ing: 

• Emphasis on exceeding standards . 
State policies that emphasize exceed- 
ing standards lead SDAs to provide 
less basic skills training and more 
classroom training in occupational 
skills for both adults and youth. For 
youth, these policies also reduce pre- 
employment/work maturity training 
and reduce the average length of 
program participation. 

• Placing greater weight on the cost 
standards . State incentive policies that 
place greater weight on cost standards 
reduce the average length of services 
for adults and increase the provision 
of pre-employment/work maturity 
training for youth. 

Finally, State performance-standards 
policies appear to have fewer effects on 
program costs than on clients or services: 

• Adjustment policies . Use of the DOL 
models has no significant effects on 
program costs, although State proce- 
dures to allow for adjustments sig- 
niHcantiy increase the amount spent 
per terminee for both adults and 
youtii. 

• Emphasis on exceeding standards . 
State policies that emphasize exceed- 
ing performance standards increase 
costs per terminee, significantiy for 
adults, although not cost per entered 



employment or cost per positive ter- 
mination. Thus, these policies lead 
SDAs to enroll less hard-to-serve 
clients but to proviae them with more 
classroom training in occupational 
skills, the most expensive service. 

• Placing greater weights on cost stand- 
ards . The weights placed on the cost 
standards do not significantiy affect 
costs per terminee or SDA perfor- 
mance on the cost standards. 

One reason that the quantitative results 
are smaller for costs may be that there is 
serious noncomparability of reported costs 
across SDAs. Some SDAs extensively 
leverage JTPA resources with funds from 
other programs, as discussed in the qualita- 
tive analysis. 

The second phase of the quantitative 
analysis examined the mechanisms through 
which the State performance-standards 
policies influenced SDA behavior. In the first 
step of this examination, we examined the in- 
fluence of a variety of SDA implementation 
practices on clients, services, and costs. In 
the second step, we examined the relation- 
ship between the State performance-stand- 
ards policies and those SDA practices that 
appear to influence clients, services, and 
costs. 

SDA policies and practices that were 
found to affect the types of clients served by 
JTPA include the following: 

• PIC influence . PIC influence per se 
does not reduce service to the hard to 
serve. The results suggest that PICs 
that see their role as guiding the 
design of the program are associated 
with greater enrollment of the hard to 
serve while PICs that are more in- 
volved in contracting are associated 
with less service to some hard-to- 
serve groups. 

• Enrollment criteria . SDAs that use ob- 
jective basic skills or educational 
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criteria serve more hard-to-serve 
clients, probably because these 
criteria are used to slot participants 
into appropriate activities. In contrast, 
SDAs that use subjective judgments, 
require previous work histories, or ac- 
cept "reverse referrals" from 
employers for OJT slots serve sig- 
nificantly fewer hard-to-serve clients. 

• Service-provider arrangements . The 
use of CBOs as service providers is 
associated with greater service to 
several hard-to-serve groups, includ- 
ing both adults and youth dropouts. 
The percentage of expenditures in per- 
formance-based contracts does not 
reduce service to hard-to-serve 
groups, although there is a weak pat- 
tern that setting stringent contract 
terms reduces the sei-vice to some 
haid-to-serve groups in JTPA 
programs. SDAs that vary the terms 
of their contracts serve significantly 
more adult welfare recipients and 
dropouts. 

• Program services . The types of 
program services offered by SDAs 
have a strong influence on the types 
of clients enrolled in JTPA programs. 
Basic skills remediation has the 
strongest association with enrollment 
of hard-to-serve groups, followed by 
classroom training in occupational 
skills, then job-search assistance. 
SDAs that provide more OJT tend to 
serve significantly fewer hard-to- 
serve clients. 

After identifying the different effects that 
these various SDA designs and implementa- 
tion practices have had on client outcomes, 
the quantitative analysis examined whether 
the practices themselves were related to varia- 
tions in State performance-standards policies. 
The findings are consistent with the overall 
relationships between State policies and 



client patterns and help elucidate the 
mechanisms by which these relationships are 
realized. 

SDAs in States that use the DOL model, 
which was positively associated with ser- 
vices to hard-to-serve clients, were less likely 
to use subjective judgments as enrollment 
criteria or to allow employers to preselect 
OJT trainees, two practices found to dis- 
courage services to hard-to-serve clients. Fur- 
thermore, SDAs in these States were more 
likely to target both youth and adult services 
to welfare recipients and groups with other 
barriers to employment 

SDAs in States with specific policies en- 
couraging services to welfare recipients were 
more likely to establish procedures to recmit 
welfare recipients for both adult and youth 
participants. On the other hand, the local 
decision of whether to target dropouts was 
not significantly affected by State policies en- 
couraging service to dropouts. 

SDAs in States with policies that em- 
phasize exceeding the standards, which were 
associated with reduced levels of services to 
hard-to-serve groups, were more likely to use 
subjective judgements about the likelihood 
of completing as a client enrollment 
criterion, which was also associated with 
reduced levels of services to hard-to-serve 
clients. These SDAs also made increased use 
of for-profit providers, which reduced the 
levels of sa^ice to some hard-to-serve 
groups. Furthermore, SDAs in States with a 
strong emphasis on exceeding the standards 
used more performance-based contracts with 
higher "holdbacks" for placements and 
higher wage rates. It is not clear from the 
quantitative analysis that these last two prac- 
tices had a direct impact on clients served by 
the JTPA program, although higher required 
wage rates did appear to discourage services 
to some hard-to-serve groups. 

SDAs in States that give greater than 
average weight to the cost standards, which 
was associated with reduced services to hard- 
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to-serve groups, also were more likely to 
allow employers to make prescreened refer- 
rals of clients for OJT slots and were less like- 
ly to use special procedures to recruit welfare 
recipients into the JTPA program. 

Overall, these results indicate that perfor- 
mance-standards policies affect the types of 
clients served, in pan through their influence 
on the types of services offered and in part 
through their influence on SDA enrollment 
and contracting practices. 

The results of the quantitative analysis in- 
dicate several State performance'Standards af- 
fect SDA decisions about whom to serve and 
the types of services to provide. These results 
are statistically reliable and have important 
policy implications. Nonetheless, the effects 
of these policies are not large. These policies 
do not preclude service to the hard to serve or 
prevent provision of intensive services. 
Rather, these policies affect SDA tradeoffs at 
the margin. 

Results from the Qualitative Analysis 

The analysis of tiie qualitative case study 
data examined how and why the 30 case 
study SDAs varied in their reactions to the 
performance standards. In tracing the effects 
of the Federal and State performance-stand- 
ards policies facing each SDA, we examined 
several elements of SDA program-design and 
implementation decisions: (1) the develop- 
ment of client goals and SDA plans for ser- 
vices to particular groups; (2) the develop- 
ment of overall service designs, including 
what services to offer, how much of each ser- 
vice to provide, how intensive to make each 
service, and how services relate to each other 
in the development of service plans for in- 
dividual clients; and (3) the design of 
management practices for selecting service 
providers and overseeing program implemen- 
tation. Within each of these program ele- 
ments, we used the case study data to address 
questions about how the performance stand- 
ards influenced SDA and service-provider 



decisions, how the performance standards in- 
teracted with other factors, and how the 
resulting SDA practices influenced clients, 
services, and costs. 

Effects on Local Program Objectives 

There was considerable variation both in 
the extent to which the performance stand- 
ards influenced local performance goals and 
in the ways that standards produced these ef- 
fects. About one-third of the case study 
SDAs had a goal of meeting the standards 
and "staying out of trouble" (that is, avoiding 
sanctions). Another one-fourth of the SDAs 
indicated a goal of exceeding the standards 
slightiy, to have a performance safety margin 
and to realize some incentive awards. The 
remaining sites indicated a goal of perform- 
ing at as high a level as they could on the 
standards, primarily because it was important 
to maxhnize their potential incentive awards. 
Pride was also mentioned by all types of 
SDAs as an important incentive to perform at 
or above the performance standards. 

The relative importance of the perfor- 
mance standards in shaping local goals and 
priorities was conditioned in part by the im- 
portance to the SDA of receiving the maxi- 
mum amount of incentive funds. Some SDAs 
placed high priority on receiving funds be- 
cause they viewed them as compensating for 
shrinking 78% allocations and tight ad- 
ministrative limits. Others valued the awards 
for their public relations value. Still others 
used the awards to undertake projects that 
they could not fiind with 78% money, such 
as projects serving high-risk clients or 
marketing or outreach projects that only in- 
directiy contributed to outcomes. 

Several typologies of responses to the per- 
formance standards help explain the varia- 
tions in how the performance standards 
influenced clients, services, and costs within 
the case study sample. Client-oriented SDAs 
gave priority to their client objectives and 
oriented their program-design and manage- 
ment decisions around how this would affect 
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their client goals. Employer-oriented SDAs 
gave priority to designing and implementing 
a service mix that would train individuals for 
jobs in demand in the local labor market. Per- 
formance-oriented SDAs gave priority to 
meeting or exceeding the perfomiance stand- 
ards (or other State or local performance 
measures). These were SDAs without strong 
employer or client goals to counterbalance 
their goals to exceed their performance stand- 
ards. 

Some client-oriented or employer- 
oriented SDAs were indifferent to the stand- 
ards when they designed their programs, 
either because they did not care if they met 
the standards or because the standards were 
so easy to meet or exceed. Others adopted 
risk management strategies that enabled them 
to meet their performance goals while still 
furthering local client and service objectives. 
Performance-oriented SDAs tended to adopt 
risk avoidance strategies that increased the 
likelihood that they would meet the perfor- 
mance standards by avoiding hard-to-serve 
clients or adjusting the service mix for 
reasons that were standards oriented. 

Finally, SDAs varied in the extent to 
which the SDA assumed responsibility for 
meeting its performance goals, shared that 
responsibility with service providers, or 
passed on the primary responsibility for per- 
formance to the service-provider level. 

In balancing these different goals, the per- 
formance standards played an important role 
in some SDAs and a minor role in others. 
Only a few SDAs in the case study sample 
purely relied on "risk avoidance" rather than 
"risk management" in their response to the 
performance standards. Most SDAs could 
meet their performance goals without having 
to resort to a major revision of other 
program goals. 

Effects on Client Goals 

The qualitative analysis suggested that 
the performance standards do not influence 
SDA client goals directly but rather influence 



clients served through their influence on 
SDA service designs and implementation 
practices. Other factors that did influence 
client goals included (1) State equitable-ser- 
vice requirements and State policies for serv- 
ing specific hard-to-serve groups; (2) the 
40% youth expenditure requirement; and (3) 
the v^haracteristics of the applicant pool, 
which were strongly influenced by environ- 
mental factors beyond the SDA's control, 
particularly the local unemployment rate. 

Moreover, the case studies demonstrated 
that the p<'rformance standards did not 
prevent SDAs that had a strong cormnitment 
to serving hard-to-serve groups from target- 
ing and serving those groups. Welfare 
recipients and dropouts were served out of 
proportion to their incidence in the popula- 
tion in a number of SDAs. Strategies for serv- 
ing these clients included (1) designing 
special training classes that integrated 
remediation and world-of-work orientation 
with occupation skills curricula, (2) leverag- 
ing other JTPA funds (for example, 8%, 6%) 
or non-JTPA funds (for example. State 
funds, foundation funds), (3) targeting 
employers that offered health benefits, and 
(4) making explicit tradeoffs to balance ser- 
vices to hard-to-serve clients with less expen- 
sive services to other groups to enable the 
SDA to meet its performance requirements. 
In addition, groups with other barriers to 
employment (the handicapped, offenders, 
and limited-English speakers) were targeted 
for services by some SDAs. These groups 
were usually served in special programs 
operated by providers with a history of work- 
ing with these groups. 

Despite this general pattern, some SDAs 
that were highly motivated to exceed their 
standards were influenced by the perfor- 
mance standards to develop "risk avoid- 
ance" strategies. These SDAs tended to ad- 
dress the needs of some subset of eligible ap- 
plicants that did not include those with the 
greatest barriers to employment. 
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Effects on Program Services and Costs 

Performance standards had slightly more 
influence on the service mix offered to adults 
by SDAs. A few SDAs mentioned the stand- 
ards when explaining their emphasis on OJT 
rather than classroom training. However, at 
least equally important influences on service 
mix were (1) the SDA's historical pattern of 
service delivery, as evolved from the CETA 
program; (2) the SDA's intentions about serv- 
ing different client groups; and (3) PIC 
preferences about what services to emphasize 
in the SDA*s service package. 

SDAs exhibited wide variation in the in- 
tensity and duration of JIT A services. In ad- 
dition, the costs of JTPA services varied 
dramatically among SDAs, and average cost 
figures often disguised tremendous variation 
within an SDA in the duration and intensity 
of services received by different participants. 
The cost standards had some influence on 
this aspect of service design, because SDAs 
often wanted not only to meet but to exceed 
these standards to maximize their incentive 
awards. For the SDAs that were highly 
motivated to "overperform" on the standards, 
the standards influenced them to design ser- 
vice programs that had short durations, 
yielded high entered-employment rates, and 
resulted in low unit costs. TTius, in these 
SDAs, the initial reaction to the standards 
was to emphasize quantity over quality. A 
number of these SDAs are beginning to shift 
toward more intensive services, however, par- 
tially in response to the implementation of 
the follow-up standards. 

However, other factors were equally as 
important in determining the intensity and 
duration of services. These included (1) the 
availability of other funds to supplement 
Title II-A funds in purchasing services to 
benefit Title II-A enroUees, (2) the cost of the 
various service packages available from the 
service providers in the SDA, (3) the SDA's 
goals about the types of clients it wanted to 
serve and the intensity and types of services 



those clients required, and (4) the desire to 
spread JTPA funds over as large a client pool 
as possible. 

There were two reasons why SDAs were 
very concerned about reducing average 
JTPA costs. A majority of SDAs visited were 
experiencing declining JTPA allocations (due 
to declining unemployment rates), which re- 
quired a decision about whether to serve a 
smaller volume of participants and keep the 
average cost of services stable or to try to 
serve as many participants as previously pos- 
sible by cutting back tlie average cost of the 
services received. Almost universally, the 
SDAs visited had decided to try to reach as 
many individuals as possible, without dilut- 
ing the service intensity beyond reason. 

The second reason for an emphasis on 
cost reduction was the SDAs' desire to ex- 
ceed the cost performance standards. Al- 
though the vast majority were in no danger of 
missing these standards, the emphasis on 
overperforming grew out of a desire to maxi- 
mize their incentive awards. 

The existence of performance standards 
for youth did not seem to affect the extent of 
SDA resources devoted to youth programs 
perse. Performance standards did, however, 
affect SDA decisions about serving in-school 
versus out-of-school youth. There is an inten- 
sive debate among and within SDAs about 
the extent to which JTPA programs should 
be offering employment-oriented or com- 
petency-oriented training. SDAs that chose 
to emphasize competency-oriented services 
found it difficult to meet their entered- 
employment-rate standards while those em- 
phasizing employment-oriented programs 
found it difficult to meet their positive-ter- 
mination-rate standards. 

Nonetheless, local goals about whether to 
run employment-oriented or competency- 
oriented programs for youth, as well as the 
Federal requirement to spend 40% of JTPA 
funds on youth, appeared to have greater in- 
fluences than the standards. However, a num- 



23^ 



Summary of Study Findings and Conclusions 



ber of SDAs had instituted a youth employ- 
ment-competency system to meet their posi- 
tive-teraiination-rate standards. 

Effects on Management Practices 

SDA management practices were 
designed to further a variety of management 
goals, including ensuring (1) that the planned 
mix of clients was actually served, (2) that 
the planned mix of services was actually 
provided, (3) that each client was referred to 
appropriate services, (4) that service 
providers enrolled clients that were ap- 
propriate for that service and had a 
reasonable likelihood of success, (5) that 
costs would be kept within reasonable limits, 
and (6) that the desired performance levels 
would be achieved. 

To further their management goals, 
SDAs instituted a number of practices for en- 
rolknent and assignment to services, place- 
ment, and selection of service providers. 
Some of these practices grew out of their ap- 
proach to the performance standards. 

Most of the SDAs visited maintained 
centralized control over the outreach, assess- 
ment, and referral of clients to service com- 
ponents, with a smaller number leaving 
responsibility for these services to service 
providers. Contrary to expectations, the en- 
rollment ofhard'tO'Serve clients was not cor- 
related with the SDA's control over 
enrollment; SDA and service-provider goals 
about serving the hard to serve seemed to be 
more important than who controlled enroll- 
ment. 

Some SDAs protected themselves against 
the risk of serving hard-to-serve individuals, 
however, by delaying enrollment until after 
some services had been delivered to avoid 
having to include early dropouts in their 
statistics. 

Performance standards did affect SDAs' 
service-provider arrangemats. In particular, 
service providers' past performance often in- 
fluenced the size of their contract in the next 
funding period. However, a number of addi- 



tional factors influenced provider selection 
and retention, often more strongly than the 
performance standards per se, including the 
availability of alternative service providers, 
the characteristics of the local economy, the 
amount and categories of services that the 
SDA contracted out, and the goals and 
philosophy of the PIC or SDA. 

Two-thirds of the case-study SDAs had 
some current performance-based contracts. 
Although the performance standards were a 
factor in the spread of performance-based 
contracting among the case study SDAs, the 
15% administrative limit was at least as im- 
portant; this was especially true in the con- 
text of declining Tide II-A allocations. 

Among performance-based and cost-reim- 
bursement contracts alike, the great majority 
incorporated performance criteria as either re- 
quirements or goals. In about half of the case 
study SDAs, contracts built in the SDA's 
standards (usually with a "cushion") direcdy. 
Most of the other half varied key contract 
terms among their providers but took care 
that, aggregating across the providers, they 
met or exceeded the SDA's own standards. 
Providers facing high placement expecta- 
tions, high wage goals, high placement 
holdbacks, and caps on maximum service 
duration were inclined to be cautious about 
the clients they would accept. If high-percent- 
age-holdback contracts become more 
prevalent, "riskier" clients, who have more 
severe employment barriers, may be more 
likely to be screened out of service programs. 

Contract terms should not, however, be 
understood as simple outgrowths or reflec- 
tions of the standards. Many of the key fea- 
tures of these contracts predate JTPA 
programs, as does the use of performance- 
based contracts in several of the case study 
SDAs. The relationship between contract 
terms and the SDA's performance standards 
was affected by several factors. These in- 
cluded how difficult a given SDA found it to 
recruit and retain clients, the relative 
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cmployability of available clients, the 
availability of alternative service providers, 
the status and direction of the SDA*s Title 
n-A allocation, its access to non-JTPA fund- 
ing, and the philosophy and goals of the PIC 
and the SDA staff. Finally, even when the 
form of performance-based contracts is a 
recent development, they often essentially 
continue the mutual expectations that an 
SDA and a provider had developed in earlier 
years. 

Service providers varied in their reactions 
to the performance expectations placed on 
them by the SDAs. In a few cases, contrac- 
tors faced pressures to perform at high levels, 
which caused greater caution in client selec- 
tion. Most providers, however, had a commit- 
ment to serving hard-to-serve clients, and 
their contract terms gave them enough 
flexibility to meet that commitment. 

For the most part, the management prac- 
tices found in the case study SDAs are consis- 
tent with "risk management" strategies 
rather than "risk avoidance" strategies. That 
is, using these practices to safeguard perfor- 
mance levels, SDAs could and did serve dif- 
ficult clients under the JTPA program while 
attaining their performance goals. 

Results of the Integrated Analysis 

Local Response to 
Performance Standards 

The JTPA legislation gives authority to 
SDAs to design programs to meet local 
needs. Consistent with this intent, this evalua- 
tion found dramatic differences among SDAs 
in the design and operation of JTPA 
programs, and many of these differences af- 
fected clients, services, and costs. Although 
perforaiance-standards policies did affect 
some SDA design decisions, many elements 
of local JTPA design reflected local condi- 
tions and constraints beyond the control of 
the SDA. The local unemployment rate in 
particular had a strong influence on the type 
of clients in JTPA programs. In tight labor 



markets, most of the clients interested in 
JTPA programs were those who had little 
work experience or major barriers to employ- 
ment, often requiring basic skills remedia- 
tion. In contrast, in areas with high unem- 
ployment, individuals seeking JTPA services 
generally had more job skills and required 
help in retraining for new industries or oc- 
cupations. The local availability of service 
providers and of alternative services also af- 
fected the types of services that were offered 
through JTPA programs, resulting in notable 
differences between urban and rural SDAs. 

Within the local context, we also found 
substantial variation in program design that 
reflected SDAs* choices of goals for the 
JTPA program. Generally, we found SDAs to 
be influenced by three different objectives: 
(1) conunitment to serving specific types of 
clients, (2) commitment to responding to 
local employer needs and interests, and (3) 
commitment to achieving specific levels on 
JTPA performance standards. The relative 
emphasis that SDAs placed on these three ob- 
jectives varied a great deal. 

To the extent that SDAs stressed client 
objectives, they tended to desigr program ser- 
vices appropriate to the needs of those clients 
and to choose management practices to en- 
sure achievement of their client goals. To the 
extent that SDAs stressed employer needs, 
they gave priority to designing and im- 
plementing a service mix to train individuals 
for the more highly skilled jobs and enrolled 
participants appropriate for that training. 

In general, most client-oriented and 
employer-oriented SDAs were able to meet 
their performance goals without major 
revisions to their other goals. In conjunction 
with clearly identified client and service 
goals, performance standards appeared to 
have their intended effects of increasing ef- 
ficiency and accountability. 

This evaluation found only a few SDAs 
that gave priority to their performance goals 
(generally goals of maximizing incentive 
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awards) at the expense of other program ob- 
jectives. Only in those SDAs without well- 
defined client or service goals did 
performance standards have notable unin- 
tended effects. In a vacuum, performance 
standards can produce unintended effects of 
reducing service to the hard to serve and 
decreasing the intensity of services. 

Effects of State Policies 

The JTPA legislation gives considerable 
authority to the States, both to establish per- 
formance-standards policies and to set 
priorities for the types of clients to be served 
in JTPA programs. There is a great deal of 
variation in how States have chosen to use 
that authority. This evaluation examined how 
SDAs responded to these differences in State 
policies. 

We found that States can provide an ef- 
fective leadership role in setting client 
priorities for JTPA programs. States that es- 
tablished target groups for JTPA programs 
did influence SDAs to enroll more hard-to- 
serve clients. Often States used performance- 
standards policies to convey their client 
priorities, including setting State standards 
for service to target groups or using 6% 
funds for service to those groups. States that 
used the optional DOL adjustment models, 
which adjust standards for the types of 
clients served, also influenced SDAs to en- 
roll more hard-to-serve clients. 

Some of the ways that States have used 
their authority in establishing performance- 
standards policies have had unintended ef- 
fects. In particular, incentive policies that 
emphasize exceeding standards rather than 
simply meeting standards tend to reduce ser- 
vice to some hard-to-serve groups and reduce 
the amount of basic skills remediation ser- 
vices. Emphasis on exceeding standards 
seems to have institutionalized sometimes in- 
tmse competition among SDAs and con- 
veyed the message that "cheaper is better." 
The effects of these policies, however, are 
not large and affect SDA choices at the 



margin. 

Effects of Federal Policies 

Federal performance-standards policies 
are intended to be neutral with respect to the 
types of clients served. They are intended to 
guide SDAs to choose cost-effective services 
but not to reduce local flexibility in design- 
ing services appropriate to local needs. This 
evaluation found that the Federal standards 
for the entered-employment rate and wage 
rate for adults generally did not have unin- 
tended effects on clients or services. Further- 
more, no evidence showed that the welfare 
entered-employment-rate standard inhibited 
service to welfare recipients. 

The Federal cost standards, however, had 
the most unintended effects and were the 
least comparably measured of all the Federal 
performance measures. This evaluation 
found that SDAs in States that placed more 
weight on the Federal cost standard tended to 
serve fewer hard-to-serve clients and that 
SDAs concerned about exceeding the cost 
standards tended to design less intensive ser- 
vices. At the same time, this evaluation 
found serious measurement problems with 
the cost standards. We found large differen- 
ces in tiie extent to which SDAs were 
leveraging JTPA funds, either by using funds 
from otiier programs to help fund JTPA Tide 
II-A programs or by using service providers 
that had alternative funding sources. As a 
result, it is difficult to compare die cost of 
services received by JTPA participants 
across SDAs. 

There was also considerable confusion 
among SDAs in how to manage youth 
programs to meet all the youth standards. Be- 
cause the goals of in-school and out-of- 
school programs were so different, SDAs 
that emphasized one type of service often 
found it difficult to meet at least one youth 
standard. Thus, die Federal youtii standards 
tended to constrain SDA choices about serv- 
ing in-school versus out-of-school youth. 
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Summary 

This evaluation found that performance- 
standards policies can influence SDAs' 
choices about clients, services, and costs. 
Some effects are the intended results of State 
leadership in setting priorities for the JTPA 
program; some effects are the unintended 
results of State and Federal policies that tend 
to reduce service to hard-to-serve groups or 
reduce the intensity of service. The unin- 
tended effects, however, are neither large nor 



inevitable. Performance standards do not 
preclude SDAs from enrolling hard-to-serve 
clients or from providing intensive services 
but do affect some SDA tradeoffs at the mar- 
gin. Furthermore, there is substantial varia- 
tion in how SDAs choose to react to 
standards. Most SDAs can meet their perfor- 
mance goals without major revision to their 
client and service goals. Only in SDAs 
without client or employer goals do perfor- 
mance standards have substantial unintended 
effects. 
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The findings of this evaluation indicate 
that the performance standards do not 
preclude SDAs from enrolling hard-to-serve 
clients or from providing intensive services. 
Nonetheless, the results of this evaluation 
also indicate that not all aspects of the perfor- 
mance-standards system are working as in- 
tended and that some policies have important 
unintended effects. In this chapter, we ex- 
amine the implications of the results for both 
Federal and State performance-standards 
policies. 

Federal Policies 

Choice of Performance Measures 
Cost Measures. The JTPA legislation 
mandates that cost standards be included as 
Federal performance standards. Both the 
quantitative and qualitative results, however, 
indicate that the cost standards had the most 
unintended effects and were the least com- 
parably measured of all the Federal stand- 
ards. There were unintended effects on both 



clients and services: States that placed a high 
weight on the cost standard lead SDAs to 
serve fewer welfare recipients and dropouts; 
in our case study sample, some SDAs con- 
cerned about exceeding their cost standards 
designed short-term, less intensive services. 

The measurement problems in the cost 
standards were also serious. SDAs that were 
able to leverage JTPA Title II-A funds with 
other program funds or that were able to rely 
heavily on service providers with alternative 
funding sources had much lower measured 
costs than those SDAs that relied solely on 
Title II-A resources to train their participants. 
Some quantitative results supported this find- 
ing, and our case studies found dramatic dif- 
ferences in the leveraging of JTPA funds. As 
a result, basing incentive payments on the 
cost standards probably rewards diffeiences 
in the local availability of other training 
resources as much as differences in manage- 
ment quality. 
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Clearly, cost-effectiveness is an essential 
goal of any program, and an increase in the 
extent of leveraging, which reduces redun- 
dant services, may be a desired effect of the 
cost standard. Nonetheless, we found that 
many SDAs were already very concemed 
about costs because of a strong desire to 
serve as many participants as possible with 
limited (and often declining) JTPA funds. 

The results of this evaluation suggest that 
alternatives to the cost standards should be 
explored. Out of concern for the unintended 
effects of the cost standards, DOL set much 
more lenient costs standards for PY 88. This 
policy is not likely to be effective, however, 
in States that strongly emphasize exceeding, 
not just meeting, standards, particularly when 
tJie cost standards are weighted more heavily. 
An altemative that requires legislative chan- 
ges would be to set maximum costs per 
entered employment (or positive iermination) 
that SDAs could spend but not to base incen- 
tive payments on cost standards. In effect, 
this policy would treat costs as a compliance 
rather than a performance-standards issue. 

Youth Measures. The only other 
problem in the choice of measures was a less 
serious concem raised about the youth 
measures. SDAs were generally confused 
about how to manage the yoath programs to 
meet both the entf^red-employment-rate and 
positive- termination-rate standards. Because 
the goals of in-school and out-of-school 
programs were so different, SDAs that em- 
phasized one type of service often felt 
pinched by at least one youth standard. One 
possible solution is to set separate standards 
for in-school programs and out-of-school 
programs so that the SDAs' choice of 
program mix would not affect their ability 
to meet the youth standards. 

A related issue is the inclusion of youth 
employment competencies in the positive-ter- 
mination measure. Although evaluating the 
quality of the competency systems was 
beyond the scope of this study, we found that 



a substantial majority of the case study SDAs 
had made a conscientious effort to establish 
meaningful systems. However, many SDAs 
reported that they would not have adopted 
the competency-based approach in the ab- 
sence of the positive-termination-rate stand- 
ard. Thus, SDAs naturally tended to develop 
systems that would reflect well on the 
programs that they were running. Despite the 
temptation to implement a superficial YEC 
system, only 2 of tiie 30 caso study SDAs 
had adopted a "quick and dirty** competency 
system, and one of those used it only as a 
back-up outcome for their employment- 
oriented youth services. 

Level of Standards 

The results of this evaluation do not indi- 
cate the need for changes in the general 
levels at which performance standards are 
set. The low levels of most standards general- 
ly allow SDAs to meet the standards and 
design tiieir programs for local needs, while 
providing a clear incentive for good perfor- 
mance. The only exception is a concem 
among a number of SDAs that tiie positive- 
termination-rate standard is too high for serv- 
ing very at-risk youth, many of whom may 
drop out of tiie program. 

DOL Adjustment Models 

The use of tiie DOL adjustment models 
appears to be very effective in ii, creasing ser- 
vice to hard-to-serve groups and in increas- 
ing the lengtii of adult services. These 
models, therefore, appear to have thek in- 
tended effects of holding SDAs harmless for 
tiieir local decisions of whom to serve in 
JTPA programs. 

The models adjust for characteristics of 
clients served and tiie length ol services, as 
well as local economic conditions. The 
models do not, however, adjust for the types 
of services offered. This evaluation has 
found tiiat tiie types of services offered sig- 
nificantiy affect the types of clients enrolled 
in JTPA programs and tiiat some of tiie unin- 
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tended effects of performance standards, par- 
ticularly the cost standards, are to reduce the 
provision of intensive services. Furthermore, 
we found that the types of services signifi- 
cantly affect program costs, with classroom 
training in occupational skills and basic skills 
remediation costing considerably more than 
job-search assistance. 

Because the adjustment models are effec- 
tive tools in holding SDAs harmless for local 
decisions, these results imply that serious con- 
sideration should be given to including some 
adjustments for program activities in the 
models for the cost standards. In particular, 
adjusting for basic skills remediation would 
enable SDAs to spend more money when 
they include basic skills remediation in their 
service package. 

Technical Assistance 

The evaluation findings suggest that addi- 
tional technical assistance from DOL could 
be used to improve the effectiveness of the 
perforaiance-standards system. First, DOL 
could provide assistance to States to develop 
their full leadership potential in setting client 
and service priorities for their SDAs and to 
develop performance-standards policies con- 
sistent with those goals. Second, DOL could 
provide additional assistance to SDAs to 
develop strategies to balance performance 
goals with other client and service goals. 
Third, although there is a moderate level of 
understanding of the adjustment model, 
many SDA respondents felt unsure that they 
fully understood how the process worked. 
Given how important the model is in en- 
couraging service to the hard to serve, DOL 
efforts to improve understanding of the 
model should increase the intended effects 
of the model. 

Reporting 

We found very little evidence that the 
Federal reporting requirements necessary to 
support the adjustment models were a burden 
to SDAs, providers, or participants. Most of 



the information required was information 
that the providers needed to manage their 
programs or for equal-opportunity documen- 
tation. Furthermore, most SDAs maintained 
computerized participant-level data bases 
(often supported by the State), so preparing 
reports on participant characteristics was not 
difficult. In fact, some respondents indicated 
that the greatest burden was the delay in issu- 
ing the JASR revisions because of the 
prolonged negotiations between DOL and 
0MB. 

However, two reporting practices reduced 
the comparability of the performance 
measures. First, typically SDAs delayed en- 
rollment of participants until after an orienta- 
tion period so that individuals who dropped 
out early were not enrolled. This pre-enroU- 
ment period usually lasted 3 to 5 days, but in 
some SDAs went on for several weeks. We 
found that SDAs with longer pre-enroUment 
periods have signiilcantly higher measured 
entered-employment rates. These results sug- 
gest that some monitoring of this process at 
the State or Federal level would increase the 
comparability of measured performance. 

Second, as discussed above, the fact that 
many SDAs leverage JTPA funds with other 
resources substantially reduces the com- 
parability of the cost standards. The problem 
cannot be rectified by reporting requirements 
because SDAs would find it very cUfficult to 
estimate the value of other resources (for ex- 
ample, the State subsidy to community col- 
leges). 

St^te Policies 

State Performance Standards 

States are increasingly adopting State per- 
formance standards to further their goals for 
the JTPA system. The study demonstrated 
that States can affect the direction of the 
JTPA program by implementing State perfor- 
mance standards to supplement the Federal 
standards. In particular. State standards for 
service to target groups furthered the Staters 
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client priorities for the JTPA program. The 
success of these policies should encourage 
other States to attempt to play a leadership 
role if they desire to do so. 

Additional Adjustment Procedures 
for the Federal Standards 

Although the effects are not entirely con- 
sistent, the quantitative results suggest that 
State policies that specify procedures to ob- 
tain additional adjustments beyond the model 
do increase service to some hard-to-serve 
groups and encourage provision of basic 
skills remediation to youth. However, in our 
case studies, we found that both State and 
SDA staff were not particularly comfortable 
with the adjustment process as it now exists. 
State staff felt unsure about how to establish 
equitable criteria for adjustments beyond the 
model and about how to determine the ap- 
propriate size of adjustments. SDA staff 
often felt that they did not understand the 
statistical basis of the models well enough to 
justify adjustments. In addition, some staff 
felt tiiat requesting adjustments was admit- 
ting failure and, particularly when SDAs 
competed for funds, that it was not fair to 
"go begging." 

One SDA director suggested an alterna- 
tive process that he felt would be more ac- 
ceptable. Specifically, he suggested that 
SDAs be allowed to request waivers from the 
performance standards for projects that met 
well-specified criteria, for example, for start- 
ing up a program for high-risk clients. This 
would be more acceptable, he said, because 
the stancards for the remaining projects 
would not be affected, so that all SDAs 
would still be abiding by the same rules. 

Hard-to-Serve Policies 

State policies to encourage service to 
hard-to-serve groups include additional State 
performance standards, integrating service to 
hard-to-serve groups into the calculation of 
incentive payments, and identifying priority 
groups. The evaluation indicates that States 



can play an effective leadership role in focus- 
ing JTPA programs on the needs of par- 
ticular hard-to-serve groups through a variety 
of incentive awards adjustments, in addition 
to the implementation of special State stand- 
ards. 

Incentive Policies 

The JTPA legislation requires that incen- 
tive payments be awarded based on the ex- 
tent to which standards are exceeded. States 
policies vary widely in how tfiat requirement 
is implemented. Policies that place a strong 
emphasis on exceeding standards are found 
to lead SDAs to reduce service to hard-to- 
serve groups, reduce provision of basic skills 
remediation, and increase the employment 
focus of both the adult and youth programs. 
These are largely unintended effects of both 
the legislative requirement and of the 
specific State policies. The results of this 
evaluation strongly suggest that these 
policies should be re-examined. 

Anotiier important aspect of incentive 
policies is the weight placed on each stand- 
ard. High weights on the cost standard were 
found to reduce service to welfare recipients 
and dropouts. States thai equally weight the 
percentage by which each standard is ex- 
ceeded implicitly give substantially greater 
weight to the cost standards because costs are 
more variable among SDAs. These policies 
should also be re-examined. 

The case studies also indicate that incen- 
tive payments are very important to SDAs be- 
cause of two other Federal policies. First, the 
allocation formula bases funding on the un- 
employment rates. Incentive payments help 
SDAs cushion die decline in JTPA funding 
brought on by an improving economy. 
Second, many SDAs are squeezed by die 
15% limit on administrative costs. Because 
30% of incentive funds can be used for ad- 
ministrative costs, these funds are particular- 
ly valuable to many SDAs. 
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Sanction Policies 

Many States have not developed specific 
policies for sanctioning SDAs for poor perfor- 
mance, and even when these policies are in 
place, Spates are reluctant to enforce them. 
The threat of reorganization has little effect 
on SDA behavior. 

Technical Assistance 

The study findings suggest that SDAs 
would benefit fi-om additional State technical 
assistance to clarify the Staters intention in 
designing its performance-standards policies 
and to discuss how the performance stand- 
ards should interact with other client and ser- 
vice priorities in furthering JTPA program 
priorities. 

Conclusions 

1 . The performance standards need to be 
balanced by local client and service 
goals to provide a useftil guide to 
managing JTPA programs. If taken as 
the sole statement of program goals^ 
the standards can lead to unintended 
effects of reducing service to hard-to- 
serve groups and decreasing the inten- 
sity of JTPA services. 
The study found that States, SDAs, PICs, 
and many service providers express strong 
support for the idea of cost-effectiveness and 
accountability for outcomes as an important 
part of the JTPA program. When SDA staff 
talk about the effect that performance stand- 
ards have had on the design and operation of 
their programs, they generally do not ques- 
tion the premise that they should be account- 
able for program outcomes. 

The idea of performance accountability, 
however, has been embraced by some in- 
dividuals in the JTPA system to the extent 
that performance goals sometimes receive 
more emphasis than other program objec- 
tives, such as client priorities and goals about 
what types of services to offer. The absence 
of clear client and service goals in some 
States and SDAs is unfortunate, because it 



can result in performance goals receiving 
undue emphasis. Performance standards had 
notable unintended effects of reducing ser- 
vice to the hard to serve and decreasing the 
intensity of services only in SDAs without 
well-defined client or service goals. 

In general, however, the performance 
standards are not preventing SDAs fiom pur- 
suing their local client and service goals and 
are not dramatically altering the types of 
clients served by JTPA programs or the types 
of services being offered. 

2. Although the performance-standards 
.system as a whole appears to be work- 
ing, the cost standards are more 
problematic than the other standards. 
Reducing the emphasis on the cost 
standards would reduce unintended im- 
pacts of the standards. 

The Federal cost standards have the most 
unintended effects and are the least com- 
parably measured of all the Federal perfor- 
mance measures. In addition, SDAs in States 
with incentive policies that emphasize perfor- 
mance in excess of the standards often find 
that reducing costs is the easiest and most 
dramatic way to perform beyond their stand- 
ards. 

Thus, concerns about exceeding the cost 
standards tend to reduce the average intensity 
of services offered by some SDAs. These 
findings suggest that alternatives that reduce 
the emphasis on the cost standards should be 
explored. 

3. State incentive policies have unin- 
tended effects and need to be re-ex- 
amined. States should be assisted in 
developing policies that enable them to 
fulfill their potential of providing 
leadership to the JTPA system. 

The State incentive policies that en- 
courage SDAs to perform as high as possible 
on all the standards have unintended effects. 
The quantitative analysis showed that these 
policies have modest but consistent effects of 
reducing service to hard-to-serve clients and 
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reducing the provision of expensive program 
services, such as basic skills remediation. 

Because these impacts are probably not 
intended, States should carefully review their 
performance-standards policies. One solution 
may be for States to establish more explicitly 
client priorities in their performance-stand- 
ards policies. Another solution may be for 
States to develop policies that are more 
neutral in their effect on SDA client targeting 
decisions and service design choices. 

Solutions should be explored to permit 
States to reward SDA performance that is 
"exemplary" rather than "average" while 
avoiding unintended effects on clients and 
services. 

4. The youth standards are currently in- 
fluencing SDA decisions about client 
targeting and program design* One 
change that would reduce unintended 
standards impacts would be to estab- 
lish separate standards for in-school 
and out-of-school youth* 
There is intense debate within and among 

SDAs about whether youth programs should 



be oriented toward immediate employment 
or toward employment-competency out- 
comes. We saw no signs of an emerging con- 
sensus on this issue. 

The performance-standards system re- 
quires SDAs to report both employment and 
employment-competency outcomes for all 
youth participating in the JTPA program. 
This compromise, however, has not been 
completely satisfactory fiiom the SDAs' 
perspective. SDAs that emphasized services 
to in-school youth often found it difficult to 
meet the youth entered-employment-rate 
standard, while SDAs that emphasized 
employment-oriented services to youth who 
have left school often found it difficult to 
meet the youth positive-termination-rate 
standard. 

Although most SDAs have developed 
program designs that enable them to meet the 
youth performance standards, there is little 
enthusiasm and considerable confusion at the 
SDA level over the youth standards. One 
change that might increase local discretion 
would be to establish separate standards for 
in-school youth and out-of-school youth. 
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The National Commission for Employ- 
ment Policy believes that this report is an ex- 
tremely valuable contribution to research on 
the Job Training Partnership Act (JTPA), and 
one whose findings have implications that go 
well beyond the scope of this project — to 
determine the impact of performance stand- 
ards on clients, services, and costs. Insights 
gained from the comprehensive analysis of 
both quantitative and qualitative data call for 
actions from national. State, and local 
policymakers that are not limited to perfor- 
mance standards. Therefore, the 
Commission's recommendations include 
both general ones as well as specific recom- 
mendations concerning changes to the perfor- 
mance management system itself. 

General Recommendations 
National Policy 

The Act balances legislative and Federal 
objectives for the program with local discre- 
tion designed to allow the program to meet 
local needs. The Commission believes that 
targeting of JTPA programs works very well 



but that the Congress must be very clear 
about who the Act is intended to serve while 
maintaining the balance between national ob- 
jectives and local discretion. Further clarity 
may be needed in defining the three major tar- 
get groups mentioned in the Act — welfare 
recipients, dropouts, and youth. For example, 
with respect to welfare recipients, since legis- 
lation affecting both the welfare system and 
JTPA*s service to welfare recipients has been 
under consideration by the Congress, it may 
be timely to clarify that long-term welfare 
recipients (those receiving welfare for 2 years 
or longer) are the welfare recipients best 
served in JTPA programs. In tfie area of 
dropouts, it may be necessary to define 
dropouts as youth (since the lack of a high 
school credential, for a youth more so than 
for adults with work experience, is a sig- 
nificant barrier to employment). For all 
dropouts, basic skills deficiencies appear to 
be more of a barrier to full workforce par- 
ticipation than educational status. 

Probably more than any other vehicle, 
performance standards have been used to 
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transmit national policy in JTPA programs. 
Sometimes these signals, transmitted through 
the choice of performance measures and na- 
tional f .andards, have not always been clear 
to State and local program managers. Recent- 
ly, much progress has been made in using 
performance standards to reinforce already 
stated policy goals and objectives. 

However, we believe that performance 
standards should reinforce policy, not estab- 
lish policy goals. The Commission recom- 
mends that the Department of Labor clearly 
articulate (1) policy objectives for the 
program apart from performance standards, 
(2) specific policy on the provision of basic 
skills for both youth and adults, (3) objec- 
tives about the intensity of training (and the 
type of training appropriate for JTPA par- 
ticipants) that are desirable, (4) appropriate 
outcomes for both in-school and out-of- 
school youth, and (5) objectives concerning 
the State role in promoting such objectives. 

The Department of Labor needs to give 
additional attention to the collection of and 
defmition of data elements to support policy 
objectives and work with the Office of 
Management and Budget to implement a 
more carefully defined management informa- 
tion system. 

State Policy 

This report has provided new insight into 
the role of States in the performance manage- 
ment system. As the report indicates. State 
policies can either promote or dissuade PICs 
and SDAs from providing intensive service 
to the hard to serve. The Commission 
believes that better use can be made of these 
policy tools through the goals and objectives 
detailed in the Governor's Coordination and 
Special Services Plan, through incentive 
award policies for exceeding performance 
standards, through sanctions for failure to 
meet performance standards, and through the 
use of incentives for serving the hard to serve 
as specified in Section 202 (b)(3)(B) of the 
Act. The Commission recommends that 



States improve their incentive policies to 
promote the provision of service to those 
clients deemed to be most in need. One par- 
ticular mechanism to achieve a clearer focus 
on the hard to serve is through better use of 
incentives for serving the hard to serve. 

Local Policy 

The Commission is concerned that one 
partner in the Act, the service provider, is 
still largely overlooked in the performance 
management system. Since JTPA services 
are often provided by entities otiier than tiie 
administrative entity, we are concerned that 
contractors need to be included in both the 
risks and the rewards of programs. The Com- 
mission recommends that PICs develop 
reward systems built into contracts for ser- 
vice providers who are successful in provid- 
ing intensive service to the most in need in 
the JTPA population, particularly for welfare 
recipients and dropouts, as well as other 
groups the PICs target for service. 

Specific Performance Management 

Recommendations 

The Commission calls attention to the 
fact that all recommendations made here are 
provided based on the performance manage- 
ment system in place during PY 86-87 
without the benefit of analyzing the effect of 
the implementation of post-program perfor- 
mance measures and standards. This report 
notes a fair degree of support for measuring 
the employment and earnings of individuals 
after they leave JTPA programs, but since no 
standards were in place during the year of 
this study, we could not directiy assess their 
effect on who is served, the type of service, 
or the cost of such service. 

We do believe, however, that most of the 
recommendations offered here will apply 
once post-program standards have been fully 
implemented into the JTPA system. 

Choice of Measures 
!• The Commission strongly supports the 
concept of cost-effectiveness inherent in per- 
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formance standards and recognized in the 
Act but believes that too much emphasis on 
efficiency measures (in this case two perfor- 
mance standards dealing with cost) may have 
had the unintended effect of precluding the at- 
tainment of effective programs to meet the 
goals of the Act. We are also concerned 
about the comparability of the cost informa- 
tion across programs. JTPA programs have 
made great strides in leveraging the resources 
of other system? to serve the economically 
disadvantaged; however, this may mean that 
the actual cost of serving some clients is 
underrepbrted in some programs. 

Therefore, the Commission recommends 
that Section 106 (b)(4), which requires the 
Secretary to prescribe performance stand- 
ards relating gross program expenditures to 
various performance measures, be amended 
to direct that cost-efficiency be monitored by 
States. 

2. The Commission endorses the concept 
that employment is the major objective of the 
Act and believes that the Act wisely em- 
phas;zes increases in employment and earn- 
ings as major goals. However, the Act also 
recognizes the importance of outcomes other 
than employment for youth — namely, the 
attainment of youth employment competen- 
cies; completion of major levels of school; 
and enrollment in other training programs. 

As the report's findings indicate, one of 
the intended effects of performance standards 
for youth that was realized was the develop- 
ment of programs to enhance a young 
person's future employability through the at- 
tainment of youth employment competencies. 
While the Commission favors this approach 
to youth programs, we are concerned about 
the continuing problems in the youth 
measures. The Conmiission endorses preserv- 
ing local programs' discretion about the rela- 
tive emphasis of serving in-school and 
out-of-school youth. The new youth measure 
introduced for PY 88 will not fully address 
the differing goals of in-school programs that 



focus on improving basic skills competencies 
and dropout prevention strategies and out-of- 
school programs that should focus on 
employability development (particularly in 
the area of basic skills) for youth for whom 
the final outcome is employment. The Com- 
mission therefore recommends that the 
Department of Labor develop separate 
reporting for in-school and out-of school 
youth programs and develop appropriate 
outcomes for both. 

Exceeding Performance Standards 
The evaluation of the effect of perfor- 
mance standards also points to another area 
of concern in the legislation that we believe 
requires remedy. Section 202 (b)(3)(B) of the 
Act directs the Governor to provide incentive 
awards for programs "exceeding perfor- 
mance standards." As indicated in this evalua- 
tion, this language may have had the 
unintended effect of promoting overperfor- 
mance and setting up competition among 
SDAs for incentive funds at the expense of 
some clients and services that might other- 
wise be offered for the hard to serve. One of 
the original principles upon which the perfor- 
mance management system was based is that 
SDAs should be judged against their own 
local circumstances, that is, economic condi- 
tions, characteristics of the population to be 
served, and so forth. The Commission recom- 
mends that the Congress clarify its intent to 
promote service to the hard to serve by 
changing Section 202 (b)(3)(B) of the Act to 
provide for incentive awards based on meet- 
ing" performance standards. At a minimum. 
States should include provisions in their in- 
centive policies that do not promote overper- 
formance at the expense of service to the 
hard to serve. 

The Commission believes that further 
discussion is needed about more specific 
guidance to the States concerning "incentives 
for serving the hard to serve." Therefore, the 
Commission recommends that an amendment 
to Section 202 (b)(3)(B) be considered to re- 
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quire States to provide such incentives. 

Encouraging the Provision 
of Basic Skills 

We are particularly concerned about en- 
couraging the attainment of basic educational 
skills for both youth and adults. We have sup- 
ported the efforts of the Secretary in em- 
phasizing the role of training programs like 
JTPA in ensuring that basic skills remedia- 
tion is a significant part of any employability 
program. The Commission is also interested 
in ways to provide incentives to JTPA 
program operators to link improvements in 
basic skills with occupational training and ul- 
timately employment. One way to improve 
the performance management system's 
ability to do so may be to collect data that 
could be used to adjust standards or provide 
bonuses for programs that emphasize the 
provision of basic skills in addition to 
employment. The Commission therefore 
recommends that the JTPA Annual Status 
Report be changed to include information for 
both adults and youth wh^, in addition to 
entering employment, attain basic skills 
while enrolled in JTPA progams (whether 
or not attained with JTPA fixnds). 

Adjustment to Standards 

The Commission is pleased that the ad- 
justment models used by the majority of 
States to establish local SDA standards have 
the intended effect of promoting service to 
some target groups for which employment 
may be more difficult to achieve (the so- 
called hard to serve); however, we are con- 
cerned that the use of any further adjustments 
does not seem to be an effective approach for 
a variety of reasons. JTPA programs must en- 
courage innovative strategies to solve the dif- 
ficult problems faced by those served under 
the Act. While we commend the Department 
of Labor's effort to promote additional adjust- 
ments, the Commission believes that it is 
time to consider alternative approaches. 

We do not believe that enough work has 



been done in this area to suggest specific ap- 
proaches, but some suggestions from recent 
Conimission publications and from the work 
of other groups should be explored. These 
studies suggest utilizing incentive approach- 
es rather than (or in addition to) the 'hold 
harmless" approach currentiy used in the ad- 
justment models. Some of the specific op- 
tions that should be considered include die 
use of incentive funds for serving at-risk 
clients based on pre-enroUment criteria such 
as targeting individuals with limited prior 
work experience, with basic skills deficien- 
cies, and who are long-term welfare recip- 
ients; and the use of waivers to performance 
standards to encourage innovative strategies 
for programs that serve the most at risk. A 
waiver could also be used for programs ex- 
pected to transcend program years, thus 
promoting longer term programs when 
suitable for the hard to serve. The Commis- 
sion recommends that the Department of 
Labor explore such alternative approaches 
which may go "beyond the adjustment 
models'' to promote positive incentives to 
serving individuals most at risk of future 
employment problems. 

Incentive Awards 

The Commission is concerned that some 
incentive award policies have become so 
complex that they may be having the unin- 
tended effect of dissuading service from the 
hard to serve simply because the policies are 
difficult to understand. This report has noted 
a number of both positive and negative ap- 
proaches currentiy in use by States that affect 
who is served, the type of service, and the 
cost of service. Translation of goals and ob- 
jectives through the use of the incentive 
award system needs to be kept as simple as 
possible to have the desired effect on local 
programs. The Commission therefore recom- 
mends that technical assistance be provided 
to States to ensure that incentive policies 
have the effect States intended and that 
States clearly reward service to clients most 
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in need. 

Technical Assistance 

The Commission is pleased that, for the 
most part, PICs and SDAs have a fairly good 
understanding of the performance standards 
system, but we are concerned that a different 
strategy in the delivery of technical assis- 
tance is needed. The Department of Labor is 
to be commended for providing assistance on 
performance standards, including their 
development and ways to adjust standards. 
However, we are concerned that thek techni- 
cal assistance on standards does not suffi- 
ciently promote the policy objectives impor- 



tant to pvstem-wide program improvement 
aimed at better preparation of the workforce. 
The Commission recommends that future 
technical assistance efforts be focused on im- 
proving programs and developing innovative 
strategies targeted to long-term enq>loyment 
of the most in need. Performance standards 
should be included as one aspect of how to 
plan and manage programs. Such a strategy 
should include PICs and service providers 
and must emphasize how to integrate 
program goals about who to serve and the 
types of service with performance expecta- 
tions into contracts. 
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1. Aims /purpose of the study 

(F. Doerfert, K. Graff, B. Holmberg, R. Schuemer & M. Weingartz) 
The aims are: 

to collect information about distance-education institutions 

- to define variables suitable for describing them 

- to analyse the relationships between these variables 
and 

- to classify institutions in groups or clusters according to 
their similarity, if possible. 



Situation at the beginning of the project 

The lack of systematic documentation of distance-education 
institutions was the starting point of the study. The situation at 
the beginning of the project may be characterised as confusing: 

Many very different organisations and institutions are engaged in 
activities in something that may be referred to as distance 
education - e.g.: 

- universities: specially designed for distance education like the 
'Open University' (UK) or distance education departments of 
conventional universities 

- .correspondence schools 

- radio and TV companies 

- private enterprises and organisations 

- governmental bodies 
and so on. 

Some of these institutions are well known and there is a lot of 
information about them available. There are others about which we 
did not know whether they were engaged in distance education at 
all. 

The project has been planned to be carried out in three steps, the 
first two of which were to be carried out by means of a written 
questionaire: 

- Step 1 : an investigation of 

study organisation 
teaching 

- media applied 

student services, tutoring and counselling 

- assessment of learners' performance 
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1. Aims/purpose of the study/ -2*^ 



The questions in step 1 were fairly general; the purpose was to 
get a first overview. Results of this step are described in: 
Buckmann et al 1985; Doerfert 1984; Graff & Kolmberg 1985; 
Holmberg 1985; Holmberg & Schuemer 1985; Neuhoff & Riechel 1985. 

- Step 2 ; replication of step 1; contacts with selected 
institutions to obtain more detailed information and further 
explanations 

- Step 3; a study of some frame or contextual factors such as 
industrialisation or national educational system 



Finally we hope to be able to develop some recommendations to 
design distance-education institutions on the basis of our 
findings . 



Some results of the first step 

Results of the first step of the project are described in the 
reports cited above. These reports contain the details. Holmberg 
(1985) gives a short summary. Here only some examples of the 
findings : 

One of the objectives of step 1 was to define some variables for 
describing or categorizing the institutions more globally; these 
variables are ^composite scores' constructed by combining certain 
items in the questionnaire. Such scores are: 

"* ^) Degree of supple mentary f ace-to->face contacts and teaching: 
the institutions differ in the amount of supplementary teaching; 
only 17 % make no use of any supplementary teaching; 49 % 
offer some form of optional face-to-face sessions; 25 % include 
compulsory f ace-to-f ace sessions and 9 % regard such sessions 
as one of the main components of their teaching, 

- b) Flexibilitvg The institutions differ in the degree of freedom 
or flexibility they give the students with regard to 

pacing their study 

- submitting their assignments whenever it suits them 

- choosing between different media 

- ^) Support vs, expectation of student independence : Some 
institutions expect and base their work on the assumed 
prevalence of students' capacity to work independently, whereas 
others start from the principle that students need help and 
support to be able to learn independently. An example of an 
institution expecting students to be independent is the 
FernUniversitaet; an example of a more supporting institution is 
the Open University. 

- d) Learner friendliness or degree of support for learners : The 
concept is based on M. Delling's term of the distance-education 
institution as a "supporting orgar ' sation ' and means the degree 
to which the institution gives support to the learners by means 
of tutoring and counselling and to which it is adaptive to 
students needs and wishes. Examples of items for this composite 
score are: 

n 



1. Aims/purpose of the study/ ''S** 



- contact with students who have not been heard from 
individualised comments on the assignments of the learners 

- short turn-around time for assignments, 

^Friendly' institutions tend to be more successful: their 'non- 
starter' rates and 'drop out' rates are lower. 



Step 2 of the study; the present stage 

Based upon the results of step 1 the questionaire was revised; 
some questions were reworded, others omitted etc. 



Parallel to the statistical analysis a documentation has been 
prepared (Doerfert & Schuemer et al 1988) containing a description 
of each of the institutions answering the questionnaire. 

This description summarises the answers to the following 
questions/topics : 

- ownership and type of institution 

- number of courses 

- subject areas of courses and educational level 

- number of learners currently enrolled 

- relative importance of face-to-face contacts 

- flexibility in pacing and teaching methods/options for students 

- media used for teaching 

- local study centres and their functions 

- evaluation of courses and media 

- elements of two-way communication 

- media used in two-way communication/counselling and tutoring 
service 

- measures to reduce the non-starter and drop-out rates 

- type of continuous assessment 

- types of items for the assignments 

- average turn-around time for tutor's correction and comments of 
the assignments 

- termination of courses with examinations and type of examination 

- success rate and non-starter rate. 
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2. Method 

2.1 Preliminary remark 
(R. Schuemer) 

The approach of the study is nomographic to some degree; it is 
assumed that there are some general characteristics or ^traits' 
which can be applied to all the different institutions; the 
recjearch team tried to express the questions and the reply 
categories in so general a manner that they are applicable to as 
many distance-education (DE) in \tutions as possible. But the DE- 
institutions are very different with respect to 

- cultural background 

- educational systems /traditions /contexts 

- structure and organisation (e,g, dual vs, single mode) 

- level of training 
etc. 

The same element may have different meanings or functions in 
different contexts or systems. Therefore it is difficult to find 
questions and categories equally relevant to all institutions. 
In spite of these limitations of the approach the method seems to 
be justified as a means to acquire basic information for want of 
systematic data on a great number of institutions. Further studies 
aiming at a more precise description of single institutions may 
use an ideographic approach to get a better fit between 
description and reality. 



2.2 The questionnaire(s) 

(R. Doerfert, K. Graff, B. Holmberg, R. Schuemer & M. Weingartz) 

The survey was done by means of a written questionnaire. Two 
versions of a (to a large degree standardised) Questionnaire were 
used: a longer version with 79 and a shor";er version with 54 
questions; the majority of the questions have several parts. (The 
additional use of a shorter version seemed to be necessary in 
order to raise the response rate - for the latter see section 2.4; 
the longer version contains all the questions of the shorter one 
with nearly the same wording and in addition some other questions 
referring foremost to the budget of the institutions.) 

Each version of the questionnaire has been worded in four 

languages: English, French, German and Spanish. 

The questionnaire is - even in the shorter version - very 

extensive and, therefore demands great efforts of those 

responding. 

The letter accompanying the questionnaire contains some 
information about the objectives of the survey. 
The front/title page of the shorter version was designed in the 
manner of TDM ("total design method' - cf. Dillman 1987): it 
contains the symbol of the FernUniversitat and the topic/theme of 
the survey in big type size - see Appendix 1. 



The questions refer to the following general topics: 

- ownership of institutions and sources of financial means 

- distribution of personnel over the departments/ delegation of 
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staff to different tasks 

- educational levels and subject areas of courses 

- number of learners 

- media used for teaching 

- regional study centres and their functions 

- face- to- face components and their importance 

- two-way communication, counselling and tutoring 

- assessment of learner performance 

- success rates, non-starter and drop out rates 

The questions in detail (the numbers in the following list refer 
to the numbers of the questions in the questionnaire; numbers 
preceded by "L" refer to the long version, those preceded by "S" 
refer to the shorter version of the questionnaire): 

- ownership and type of institution (L2, S2) 

- aims of the institution (L3) 

- annual budget in US $ (L4) 

- sources of financial means (L5) 

- distribution of financial means over the departments of the 
institution ( L6 ) 

- personnel costs in relation to total costs (L7) 

- distribution of personnel costs among departments (L8) 

- services rendered by own institution and services rendered by 
other institutions (L9) 

- open-ended question referring to the meaning of the terms 
^programme', 'curriculum', 'course' (LlO, S3) 

- number of courses, programmes and curricula (Lll, S4 ) 

- educational level of courses offered (L12, S5): 

— I: basic school education 

— II: further education at school and basic professional 

tra ' ning 

— Ill: university study and further professional training 

— IV: others 

- subject areas covered by the courses offered (L13, S6) 

- langii£ige used in study material (Ll3a, S6a) 

- number of learners currently enrolled: total number (L14, S7) 
and nvitiber per subject area (L16, S7a) 

- average time needed to acquire a degree or certificate (L15) 

- minimum conditions to be complied with in order to study at the 
institution (e.g. minimum qualifications ^r payment of fees) 
(L17) 

- number and type of degrees /diplomas offered (LIB) 

- number of research projects related to distance education (L19) 

- restrictions as to the intake of students and the employment of 
staff (L20) 

- number of staff members employed in the different departments of 
the institution (L21) 

- importance of the computer for different tasks (L22) 

- relative importance of face-to-face contacts (L24, S8) 

- flexibility in pacing and teaching methods/options for students 
(L25, S9) 

- cooperation with other institutions (L26) and its importance 
(L27); use of study material and other services offered by other 
institutions (L28) viz. services rendered to other institutions 

/ (L29) 

ij - media used for teaching the different subject areas (L30, SIO) 
ERJ^C - prescription of media combinations (L31, Sll) 
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- share of different media in the media combinations (L32, S12) 

- course teams and their participants (L34/35, S13/14) 

- media specialists for course development: members of staff or 
outside personnel (L36, S15) 

- local study centres and media used there (L37, K16) 

- functions of the local study centres: face-to-face teaching 
compulsory for students vs. support of distance learning 
(voluntary participation) (L39, S17) 

- evaluation of courses and media (L40, Sl8) 

- research into the conditions and methods of distance education 
(L41) 

- guidelines for selection of teaching media (L42, Sl9) 

- estimate of the expenditure for media - relative to the total 
budget (L43, S20) 

- number of staff members mainly working on media development 
(L44, S21) 

- flexibility of curricula (L45^ S22) 

- opportunities for learners to design their study programmes 
tailored to their individual needs / use of learning contracts 
(L46, S23) 

- main aims of distance teaching at the institution / aims of 
self-checking exercises and examinations (L47-49, S24-26) 

- encouragement of study groups or learners' self-help qroups 
(L50, S27) ^ 

- responsibility for the assessment of learner performance (LSl^ 
S28) ^ ^ 
elements of two-way communication (L52, S29) 

counselling and tutoring service /media used in two-way 
communication (L53 ^ S30 ) 

times for contacting the counselling services / also after usual 
office hours (L54, S31) 

percent of learners using the counselling services (L55, S32) 
encouragement of learners to contact their tutors whexi they feel 
they need help (L56^ S3 3) 

measures to reduce non-starter and drop-out rares (L57/58, 
S34/35) ^ 
if a learner is enrolled for several courses: is he assigned one 
central tutor for all courses or are there different tutors ? 
(L59, S36) 

reference to one central counsellor or several counsellors for 

different problems areas ? (L60, S37) 

types of continuous assessment (L61, S38) 

main purpose of the assignments: assessment of learner 

performance vs. learner support (L62, S39) 

termination of courses with examinations / type and function of 
terminating exams (L63/64, S40/41) 

relative number of course units including/prescribing 
assignments (L65, S4 2) 

submission of assignments whenever it suits the learner (L66, 
S43) 

average time lag (in weeks) between two assignments in a course 
(L67) 

average turn-around time for tutor's correction and comments of 
the assignments (L68/ S44) 

types of items for the assignments (L69, S45) 
commenting on assignments and extent/amount/quantity of 
commenting (L70/71, S46/47) 

individualisation of tutor comments (L72, S48) 

?5 
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- responsibility of correctors of assignments for other tasks 
(e.g. answering student's questions or counselling ) (L73/ S49) 

- manner of payment of correctors /tutors (L74/ S50) 

- use of computers for correction and commenting of /on assignments 
(L75, S51) 

- success rate and non-starter rate for the different subject 
areas or groups of subject areas (L76/ L79/ S52) 

- success rate and non-starter rate for the three courses with the 
highest number of enrolments (L77/78/ S53/54) 



2.3 Dispatch 

(F. Doerfert, R. Schuemer, C. See-Bogehold & C. Tomaschewski) 
The dispatch was done in three steps: 

1) first dispatch of the long version of the questionnaire in 
August/ September 1986 

2) second dispatch of the long version of the quectionnaire 
together with a reminder in Februrary/March 1.987 

1) first dispatch of the short version of the questionnaire in 
August/September 1987 



In step 1) the questionnaire was sent to 1640 addresses; some of 
these addresses were not those of distance education institutions 
but of other bodies or individuals; it was hoped that these 
institutions (inclusive of minietries of education) and interested 
persons would function as multiplicators for the distribution of 
the questionnaires or would have some useful contacts in their 
countries . 

92 of the 1640 addresses could not be delivered. 626 belonged to 
institutions which are no longer or not yet engaged in distance 
education or are research rather than teaching institutions. 922 
addresses of institutions remained. These were the data base of 
addresses for the dispatch in step 2) and 3). The addresses of the 
institutions that returned a completed questionnaire in step 1 
were eliminated from the data base for step 2) and 3). (The 
regional distribution of addresses for the dispatch is summarised 
in Table 1; for information on the response rates in the different 
regions see Table 3 under section 3.1 .) 

In countries for which we did not have a version of the 
questionnaire in the national language the most similar language 
version or a version in the most probable second language, 
respectively, was used, for example: 

- Eastern Europe: English and German (exception: Romania, where a 
French version was used additionally) 

- Portugal: Spanish and English 

- The Netherlands and Scandinavia: English and German. 

(Which language(s) was (were) used for each country can be seen 
in Table A_l in Appendix 3 to section 3.1) 
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Table 1 s Regional Distribution of the addresses for the first 
veSfon (step^!)^°''^ ^^^^^ °f 

He gion ~li5ncr I ge^ul^c?"- 
Australia 72 
Asia 

Africa 93 

South America (incl. 
Central America and 180 
the Caribbeans) 

North America 328 
Western Europe (incl. 
Scandinavia) 651 
Eastern Europe (socialist 
states) 125 



40 
109 
49 



85 
154 
210 
94 

Sum 1.640 741 



2.4 On the state of the data/ some problems and possible errors 
(R. Schuemer & C. Tomaschewski ) 

The results presented under 3 may contain some errors. Some 
reasons for these are: 

- the questionnaire is standardised to a great degree, but it 
contains some open-ended questions; independent coding of the 
?!;pr^?nr-o°t.^ r questions showed low inter-rater concordance; 
Therefore the frequencies for the answers to the open-ended 
questions are not reported here. 

The coding of the answers to the other questions is also 
susceptible to error, however. There are various reasons for 

— Not all questions and categories for the answers seem to be 
Clear; on the contrary, some respondents' comments point to 
misunderstandings caused by some of the wordings in the 
questionnaire. 

— Several of the institutions are dual-mode institutions. It i 
not evident in some cases whether the answer to a question 
refers to the institution as a whole or to the distance- 
education department of the institution. 

— The answers to related questions are contradictory in some 
cases; answers to questions at the beginning of the 
questionnaire conflict with later answers (e.g. different 
subject areas are mentioned in answers to different, but 
related questions). 
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— In some cases comments on categories ticked off conflic:t with 
the chosen category. 

— Some respondents seem to have changed their 'strategy' of 
answering during the process of filling in the questionnaire 
- probably a result of the too voluminous questionnaire and 
its complexity. 

Some of these ambiguities could be cleared up by considering the 
context and other information material (e.g reports on/from the 
institution, study guides etc.); but such additional information 
was not available in all cases. 

- Further coding problems stem from the attempt to transfer 
related terms from one educational system to another and in 
addition by translating from one language into another (it may 
be preferable to use only one language version^ > in such 
projects since the questions and answers might be more 
comparable - even if this may produce lower response rates in 
countries with a different language). 

- Another problem may result from combining data from the long and 
the shorter version of the questionnaire. Since the common 
questions in both versions have nearly identical wordings and 
their logical structure is the same in both versions, this may 
be a minor problem - see section 2.5 for the details about the 
combining of the two versions. 



2.5 The transformation of the data from the long version into the 
data format of the shorter version 

(F. Doer f art, R. Schuemer & C. Tomaschewski ) 

Answers to questions with different numbers in both versions but 
identical content and wording were transferred directly into the 
common datafile. 

The categories for subject areas in the long version are more 
differentiated than those for some questions in the shorter 
version. In this case it was necessary to combine some categories 
of the long version to yield data comparable with those for the 
corresponding questions in the shorter version. This was done by 
combining categories '(1) Education..., (2) Humanities, music and 
the arts and (3) Languages...' from the long version and by 
subsuming them under category '(1) Education, the humanities, 
music and the arts, languages' of the shorter version, just as 
long-version categories '(4) Law... and (10) Social sciences...' 
were transformed into the short-version category "(2) Social 
sciences and law' and the long- versions categories '(7) 
Engineering and technical professions, (8) Sciences (9) 
Mathematics' into the short-version category "(5) Mathematics, 
sciences and engineering'; data referring to the long-version 
categories '(5) Economics...' or '(6) Agriculture...' or '(10) 
Medicine...' were assigned to the corresponding short-version 
categories (3) or (4) or (6), respectively. 

Different mathematical operations were used for different 
questions in this process of combining categories of the long 
version: 



1) in a widely used language such as English 
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o»J "^T^^^s °^ learnsrs per subjact area (L16 long version) were 
smmed up over the corresponding categories to produce Jhe 
figures m short-version format (STa)! y^^auce tne 

Percentages (relative frequencies -.e.g. the success rat-Ps in 

For questions asking the respondent to tick off categories fe o 
L13: subject areas for different educational levels) ?Se ^ 
categories were combined by logical ^OR' : if one or more of the 

i?ck;roff°^Sf T"'"' ^'"^^r" ^° combined :e?e '''^ 

ticKed off then the corresponding category of the short version 
was defined to have been ticked off=>. version 
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a o5\JrShn2rf^^°? °^ ""^^ i^to the format 

tKJL ,, ,,.J^^^ 'if/- and ^assignment statements' \ 
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3i Results 



3,0 Overview/Prenote 

The following representation of the results is descriptive and has 
mainly a summarising function. In this context we refrain from 
interpretation . 

Section 3.1 contains some data on the response rate. 

Section 3.2 describes some general characteristics of the 
institutions (type, size, educational level, etc.) and contains 
some information on their budget (sources of budget and its use). 

Section 3.3 deals with teaching/ learning methods. 

Section 3.4 describes the results for teaching media. 

Section 3.5 is concerned with two-way communication (tutoring, 
counselling and assessment of learners' performance). 



20 



3.1 Response rate/ "12" 



3.1 Response rate and regional distribution 
(F, Doerfert, R. Schuemer & C. Tomaschewski) 

T^nH^'Si? K^^^S"" ^5 i^^ternational surveys directed to institutions 
(and not to individuals) is the low response rate. There are 
several reasons for this - for example: 



- postal problems (e.g. in developing countries) 

- bureaucratic or political problems (e.g. in most of the Easter 
European countries) 

- the complexity of the topics/ sub jects of the investigation 

- the great variety of institutions which makes it difficult to 
express the questions in such a way that they seem relevant to 
all the respondents . 



Therefore, one cannot expect a high overall response rate. 

There was a total of 922 addresses of distance education 
institutions after eliminating 

- those institutions which are not (or no longer) engaged in 
distance education or ^ ; y y 

- those addresses which are not correct (postal delivery not 
possible - see section 2.3 above). 

215 filled-in questionnaires were returned; 18 of these 
questionnaires were disregarded because of uncompleteness (a great 
number of questions not answered). So the sample contains 197 
institutions. 122 of these institutions answered the long version 
and 75 answered the short version of the questionnaire. 

"^^^m^u^^^o"^^®^ °^ responses for different regions are summarised 
in Table 2 (The corresponding data for each country are presented 
m Table A_l in Appendix 3). jr f 



Table 2: Regional distribution over continents/ political 


regions | 


(a) 


dispatch 


(b) number of responses 






(a) 


(b) 
. 14 


fa) 

Latin America 

(incl. the Caribbean) . 126 . . 








. .39 




. 119. . 


. .17 


Western Europe 
(incl. Scandinavia) ... 269 . . 








..53 


Australia 

Northern America 
( USA + Canada ) . . 


. .50. . 
.201. . 


. .20 
.51 


Eastern Europe 
( socialist countries ) . . 92 . . 


. . .3 




Total 


: (a) 


922 and (b) 197 
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3.1 Response rate/ "13" 
The resulting response rates are shovm in Table 3r 



Table 3: The response rates 



overall: 21.4 % 



- Africa: 21,5 % 

- North America: 25,4 % 

- South America: 31.0 % 

- Western Europe: 19,7 % 

- Eastern Europe: 3,3 % 

- Asia: 14,3 % 

- Australia: 40.0 % 



This overall response rate of 21 % is higher than that of the 
preceding step 1 of this study where a response rate of 14.3 % was 
yielded (see section 3,4 in Holmberg & Graff (eds,) 1985); but the 
representativity of the results is very doubtful. 
Therefore > any generalisation with regard to the 'population' of 
all distance education institutions must be regarded with some 
reservations - strictly speaking, the frequencies and relations 
described in this report are valid only for the sample, 

A second fact beside the low response rate should cause any 
attempt at generalisation and interpretation to be cautious: this 
is the uneven distribution of the response rates over the 
countries or regions (see Table 2 above: The highest response rate 
is for Australia, followed by South America; the lowest response 
rates concern the socialist countries in Eastern Europe^>), it is 
obvious that, for example, the very low response rate of the 
Eastern European countries forbids any generalisation from the 
sample to the distance-education institutions in these countries. 

Furthermore there are reasons to assume that state-owned 
institutions (e,g, in the U,S,A, ) are over-represented and private 
institutions under-represented in the sample (see below). 



1) There are also great differences between the countries within 
the continents; for example, the relatively high response rate 
for South America results mainly from the replies emanating 
from only two countries: Colombia and Argentina, See Table A_l 
in Appendix 3 for the details, 
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3.2s General characteristics of the institutions/ '•14' 
3.2 General characteristics of the institutions and their budgets 



(R* Schuemer) 



Questions in the short version are specified by ^Q' followed by 
the question number; questions in the long version are specified 
by 'L' and the number. 



3.2,1 Age of institutions 



The great majority (almost three quarters) of the institutions in 
the sample were after World War II (1. quartile « 1944; - see 
Table 4) 



Table 4: Founding year of the institution 



- range: from 1827 to 1987 

- median: = 1967 

- first quartile: 1944 

- third quartile: 1975 



N= 185; 12 X "no answer 



3, 2 ,2 Ownership and aims of institutions 

About one half of the institutions are state-owned (Q2; see 
Table 5) . 



Table 5: Ownership and type of institution (Q2): 



- state-owned 95 

- official body under public law 34 

- private, commercial institution 27 

belonging to an association of some 
kind (trade union, church, etc.) 24 

- other type of institution 24 



N=196; 1 X "no answer" 



There are a few institutions with multiple responses; therefore 
the number of nominations (204) is greater than the number of 
respondents (N=196) . 



The long version of the questionnaire contains a question about 
the aims of the institutions (L3) and their importance. 
The respondents had to rate the aims on a 5-point scale ranging 
^ from "0": "no importance" to "4": "very much importance". Putting 
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together the frequencies for the categories "3" ("much") and "4 
("very much importance") results in the order of aims shown in 
Table 6* 



Table 6: Aims of the institutions and their importance (L3): 

fa^*: frequency of answers "3: much" + "4: very much 
importance" 



f 3-^4 



- offer of further training opportunities 
(recurrent education): 81 

- opening of school/university to new 

target groups: 72 

- introduction of new teaching/learning media 

and technologies: 65 

- introduction/application of new teaching/ 

learning strategies: 64 

- increase of available number of places for 
students : 59 

- extension of the number of subjects on offer: 44 

- reduction of costs in the educational sector: 44 

- profit: 14 

- others : 19 



N=120; 2 X "no answer" 



The comparatively high number of state-owned institutions (55 of 
122 institutions responding to the long version of the 
questionnaire) may explain why ^profit' plays a minor role in the 
order of aims. 



3.2.3 Educational level and subject areas of courses 

It is very difficult to compare educational levels and school or 
university grades or degrees in countries with different 
educational systems and traditions. The resaerch group tried to 
find wordings widely applicable to different educational systems 
(see Q5) - but it is obvious that the short descriptions used in 
the questionnaire suffer from vagueness to some degree. The 
categories are very rough and not unequivocal. Therefore the data 
should be interpreted with caution. 

The interpretation of the data also has to take into account that 
one institution may offer courses for more than one educational 
level (many institutions gave multiple responses to Q5). Table 7 
summarises the data for the total sample; 
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Table 7; Educational level (Q5) of courses offered (Q5a): 



basic school education (up to 15/16) 53 

further education at school and basic professional 
training (e.g. preparation for ^ A' levels, high school 
certificate and entrance qualifications for university 
studies, upper secondary level, technical college etc.). 

university study and further professional training 
(after basic professional training and/or job experience 

others 



yes no 
143 



101 


95 


152 


43 


35 


159 



The following question (Q6) gives some information as to the 
subject areas of courses offered. Table 8 summarises the data for 
courses of level III (university study and further professional 
training) 



Table 8: Subject areas of courses - level III - (Q6): 
frequencies of nominations 

(1) education, the humanities, music and the arts, languages ... 68 

(2) social sciences and law 33 

( 3 ) economics ............ .25 

(4) agricultural sciences, agricultural and sylvicultural 
professions n 

(5) mathematics, sciences and engineering .'.'28 

(6J medicine and medical jobs and professions !!!ll 

( 7 ) others : ! .' ! 12 



The majority of institutions (152 from 197) offer courses for 
level III ("university study and further professional training") 
according to Q5), but only 82 of the 152 institutions ticked off 
one or more of the subject areas in Q6 for level III. 
Out of these 82 institutions only 43 offer courses in at least two 
subject areas on level III, and 30 offer courses in at least three 
subject areas (only 19 x four or more subject areas). 

Taking into account all information available we tried to classify 
the institutions into the categories 'universities' and 'non- 
universities' ('university' here in the (Western) European meaning 
of the term): according to this provisional classification 72 of 
the institutions offering level Ill-courses can be classified as 
'universities ' . 



Table 7 and Table 8 are based on the data for the total sample 
(N=197). 

The groups of subject areas in Q6 are rather broad; it is possible 
to arrive at a more differentiated picture of subject areas 
covered by distance education by regarding the data for the 122 
institutions responding to the longer version of the questionnairv? 
(L12, L13 - see Table 9 and Table 10). 
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Table 9: Educational levels (L12) 



- basic school education; 40 

- further education at school and basic 
professional training: 66 

- university study and further professional 
training: 91 

- other 22 



N=122; 1 X "no answer" 



Table 10: Subjects areas (L13) for level III (university study) 



(subjects areas ordered by the frequency of mentions: fy««) 
subject area fy*« N 



1 
5 
2 

10 
3 
4 
7 
9 
8 

11 
6 

12 



education, teacher training: 58 112 

economics : 57 113 

humanities, music and arts: 46 113 

social sciences: 46 113 

languages, linguistics: 42 113 

law and legal professions: 37 113 

engineering and technical professions: 31 113 

mathematics: 32 113 

sciences: 29 113 

medicine and medical jobs: 20 113 

agriculture, sylviculture, forestry: 15 113 

others: 22 113 



The comparatively high number of nominations for "education" 
reflects the fact that several institutions are engaged in 
trciining and further education of teachers. 



In view of the differences in educational levels between the 
institutions it is not surprising to find that there is a wide 
range of conditions learners have to fulfill in order to study 
(see Table 11). Beside ^payment of fee' (94 x) the condition 
mentioned most frequently is ^minimum qualifications' (66 x; N=119 
respondents; 3 x "no answer" [question L17 in the long version of 
the questionnaire] ) . 
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Table 11: Conditions to be complied with in order to study (L17): 

N number of answering institution^^ 

fymm* number of institution mentioning the condition 
% : percentage related to 361 nominations in replies to this 
question (sum of fy«« over conditions) 



N fy % 



— simply ordering course materials: 119 41 11,4 

— immatriculation? 119 56 15,5 

— payment of fees: 119 94 26,0 

— learning contract with the institution: . 119 28 7.8 

— continuous assessment: 119 49 13.6 

— minimum qualifications (e.g. minimum age, 
entrance exam, job experience, 

minimum grade) : 119 66 18.3 

-~ others: 119 27 7.5 



3.2.4 Teaching modes (face-to-face vs distance teaching) and 
study centres 

Institutions differ in the degree to which face-to-face components 
are part of the teaching method; furthermore in some of the 
institutions face-to-face components are voluntary while in others 
they are compulsory. 



Table 12: Teaching mode/relative importance of face-to-face 
contacts (Q8 ) : 



a) pure distance teaching, no face-to-face contact 51 

b) distance teaching with very little face-to-face contact 
(for tutoring/counselling purposes or immediately before 
oral examinations 53 

c) distance teaching with a few face-to-face elements or 
supplements (e.g. seminars, week-end seminars etc.) 86 

d) distance teaching and face-to-face teaching of equal 
importance 28 

e) face-to-face teaching with supplementary distance teaching 
material 25 

f ) others \ 5 



(N=192; 5 X "no answer") 



face-to-face sessions, if offered: voluntary .. 58/ compulsory . .46/ 
partly, partly.. 29/ "no answer".. 64 
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26 institutions have given multiple responses (probably indicating 
that different methods are used for different courses, programs or 
curricula). The combinations and their frequency are: 

- category a ) and c ) : 3 x 

- category b) and c): 10 x 

- category b) and e): 2 x (categories a) e): 

- category c) and d): 2 x see Table 12 above 

- category d) and e): 3 x 

- more than two categories: 6 x 



Putting together b) and c) in one category and disregarding the 16 
institutions with multiple responses (others than b)+c) ) yields 
the following distribution: 

- 39 ;pure distance teaching institutions' without face-to-face 
contacts 

- 99 distance teaching institutions with a few face-to-face 
elements 

- 14 institutions with distance teaching and f ace-to-f ace teaching 
of equal importance 

- 14 institutions offering face-to-face teaching with 
supplementary distance teaching 

The face-to-face components are compulsory at 

- 26 of the 99 institutions with only a few face-to-face elements 

- 4 of the 14 institutions with distance teaching and face-to-face 
teaching of equal importance 

- 7 of the 14 institutions offering face-to-face teaching with 
supplementary distance teaching. 

Face-to-face contacts take place frequently in regional study 
centres; about one half of the institutions have such study 
centres (see Table 13): 



Table 13: Study centres and their function (Q16/17): 



- study centres: ves/no 

"yes": 99 

"no" : 94 

"no answer" : . . 4 

- function of study centres : 

- support of distance teaching (voluntary participation): 72 

- compulsory face-to-face teaching: 54 

- others : 26 



There is a tendendy that the study cen\:res have rather the 
function to support distance study. 
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3,2 • 5 Flexibility: Options for learners 

The institutions differ in offering options to their learners with 
regard to pacing their studies and to media selection (see 
Table 14) . 



Table 14: Flexibility in pacing and teaching methods /options for 
students (Q9) ; 



f : frequency of mentioning 
N : number of respondents 



Flexibility (in): N 

- when to start their course/their studies 105 182 

when to take their exams/final exams 92 182 

- when to order their study material 92 182 

- when to submit their assignments^ > 96 182 

- between different teaching methods 32 182 

- between different media 4i 181 

- how and when to use counselling services 123 181 

- others 19 182 



One can define a flexibility-score (FX) for each institution by 
scoring each mention of a category as 1 (else 0) and summing up 
over the categories; the resulting score simply gives the number 
of options offered to learners by the institution. 

- 30 institutions offer no options (FI=0); included: the 16 
institutions with "no response" to Q9 

- 23 institutions offer only one option in the sense of Q9 (FI=1) 
and 34 only two (FI=2) 

- 27 institutions offer three options (FI=3) and 31 offer four 
(FI=4) 

- 32 institutions offer five options (FI=5) and 20 offer more than 
five options (FI>5) 



Many institutions are also flexible with regard to the curricula 
they offer (see Table 15): 



Table 15: Flexibility in the curriculum to be followed for the 
acquisition of a particular diploma (Q22): 

Flexibility: 

- low: fixed curriculum 84 

- medium: fixed curriculum , but it contains alternatives ... 92 

- high: courses in the curriculum can be chosen freely 
within a set framework c 39 

- others : 2 



N=:183; 14 X "no answer" 



1) Question 43 (similar to Q9) yields a somewhat higher rate of 
institutions allowing their learners to submit assignments 
% whenever it suits them (see section ^-j^W 
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3.2.6 Size of institutions: nximber of learners and number of 
courses 

The 'size' of an institution can be indicated by the number of 
learners, the number of staff members, and the number of courses, 
programmes, and curricula. It can also be expressed by its budget. 
In this section only the number of learners and the number of 
courses will be considered (see 3.2.7 for budget and staff). 

The number of learners varies to a great extent (from 10 up to two 
millions) 2 > ; about one quarter of the institutions has only 1,000 
or less learners; about one half of the institutions has up to 
3,000 learners; and only one quarter of the institutions has more 
than 10,700 learners (see Table 16). 



Table 16: The number of 


learners (Q7 ) 


range: from 10 to 2,000, 


000 


first quartile 1,000 




median ^ 3, 000 




third quartile - 10,700 




"no answer" : 9 x 



We tried to differentiate this overall number of learners with 
regard to the subject areas; Table 17 shows the median of the 
number of learners enrolled in each subject area. The interpre- 
tation of the data in Table 17 has to take into account that the 
majority of the institutions offer courses in only one or two 
subject areas; therefore the frequency of institutions responding 
with learner numbers greater than zero is rather small for each 
subject area. The medians in Table 17 refer to these institutions. 
The greatest number of learners - not regarding the residual 
category "(7) others" - can be found for "(1) education..." (Q7a: 
Median: near 1,000). 



2) "2,000,000" is an estimated number for the audience of 

programmes produced by ORTS, a radio station with educational 
programmes in Dakar, Senegal. The next lowest number is 141,212 
From other sources we know that, for example, the corresponding 
teaching Central Radio and Television University of China in 
1988 has over 700,000 students registered for degree courses 
and that the Sukhotai Thammathirat Open IJniversity (STOU) in 
Thailand has more than half a million students-. 
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Table 17: Number of learners for each group of subiect areas 
(Q7a): 



N : number of institutions reporting number of learners 
greater than zero for a subject area 

Med:>o : median for those institutions reporting number of 

learners greater than zero, 
fo : number of institutions reporting a number of "0 learners 

for a subject area 



Med:.o N 

(1) education^ the humanities^ music and the arts. 



languages 1,000 101 79 

(2) social sciences and law 558 65 114 

(3) economics 56? 61 118 

(4) agricultural sciences, agricultural and 

sylvicultural professions 200 21 159 

(5) mathematics, sciences and engineering 698 68 112 

(6) medicine and medical jobs and professions 209 21 161 

(7) others: 1,445 47 132 



One gets a more detailed picture from the responses to the long 
version where the subject areas are more differentiated (see 
Table 18); but it has to be taken into account that the numbers 
of institutions responding for each subject area are even smaller. 
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Table 


18: Number of learners for the 


different 


subject 


areas 




(L16) : 








o • 


number of institutions reporting numbers 


greater 




than zero 










median for those institutions 


reporting numbers 


greater 




than zero 














Med>o 


Range 


- ( 1) 


Education, teacher training:.. 


42 


798 


10-20,000 


- ( 2) 


Humanities, music and the arts 


: 30 


560 


4 -8,122 


- ( 3) 




26 


318 


5-21 ,313 


- ( 4) 




20 


146 


17-18,288 


- ( 5) 






558 


54-60.000 


- ( 6) 




12 


200 


8 -7,000 


- ( 7) 


Engineering and techn. profess 


. : . . .29 


400 


20-12,000 


- ( 8) 




27 


450 


8-11,010 


- ( 9) 




26 


600 


12-15,404 


- (10) 




33 


471 


15-26,439 


- (11) 






402 


45-11,000 


- (12) 




25 


2,000 


50-50,407 1 



The number of courses, programmes and curricula can be regarded as 
another indicator of the size of an institution (see Table 19). It 
should be considered, however, that terms like 'course' or 
'programme' may have different meanings in different institutions 
(although we tried to give a short definition of the terms in the 
questionnaire); in addition, courses may contain many course units 
in some institutions while they may comprise only a few units in 
others and the units may differ considerably with regard to their 
length . 
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Table 19: The number of courses, programmes 
and curricula (Q4): 



- number of courses 

range: from 0 to 4000; 
median 49 courses 
(N=172; 25 x "no answer") 

- number of programmes ; 

range: from 0 to 363; 
median ^ 2; 

(N=167; 30 x "no answer") 



- number of of curricula ; 
range: from 0 to 600; 
median ^ Z} 

(N=167; 30 x "no answer") 



The number of members of staff is another indicator for the size 
of institutions. Question 21 (L21) in the longer version of the 
qaiestionnaire gives some information about this. The answers of 
the 106 institutions responding to the question vary to a great 
extent; the numbers for the members engaged in teaching vary from 
0 to 2,000 (see section 3.2.8 and Table 26 below for the details). 

3 •2, 7 Budget and sources of means 

Only the longer version of the questionnaire contains some 
questions referring to the financial means and their distribution 
(L4-L8) . 

Table 20 shows some data on the annual budget. 



Table 20: Estimated size of the annual budget in US $ (L4) 
frequency (f) and percent (%) for the intervalls: 



% 



- up to 500,000: 42 (37.5%) 

- 500,000 - 1,000,000: 15 (13.4%) 

- 1,000,000 - 10,000,000: 32 (28.6%) 

- 10,000,000 -50,000,000: 17 (15.2%) 

- more than 50,000,000:... 6 ( 5.4%) 



(N=112; 10 X "no answer") 

Median: between 500,000 and 1 million. 
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The answers on question L4 give only a rough estimate of the 
annual budget in US $ . Regarding the data one has to take into 
account : 
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• The rate of exchange of US $ in relation to other currencies 
varies * 

- Costs of living and personnel costs are hardly comparable in 
different countries. 

Perhaps more expressive than the absolute quantities like those of 
Table 20 may be the figures about the relative importance of the 
sources of financial means (Table 21) or the distribution of 
expenses over the different departments or tasks within the 
institution (Table 22). 



Table 21: Sources of the financial 


means at the disposal 


of the 


j.ns uj. uunon 












N. number of respondents 












fxi frequency of institutions 


with 


the answer "x 








instituions with 






cixio wc;x o o u T> ^1. nxgneL^ ) 












Source 


N 


range 


fo 


f a:50 


£a:80 






0-100% 


28 


48 


32 


- sales profits from teaching 












material, income from 












searvices to other 
















0- 80% 


67 


2 


1 


- interest from (foundation) 
















0- 10% 


94 


0 


0 


- subsidies /donations from 












private persons or bodies:. 


.111 


0-100% 


90 


4 


2 


- subsidies from owner or 












umbrella organisation (with 












the exception of the state) 


;111 


0«100% 


91 


6 








0«100% 


42 


43 


27 






0-100% 


87 


3 


2 


Median for "learners' fee": 


between 33 and 


35%; 






median for ^ state subsidies' 


: 16 % 


; 








median otherwise: near zero 













It is evident from Table 21 that the main sources of the financial 
means are learners' fees and state subsidies; with regard to the 
latter it has to be taken into account that 55 of the 122 
institutions responding to the longer version of the questionnaire 
are state owned. 
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Table 22: Estimated percentage of financial means for each 
department (L6 ) : 



N/ to, fa:50/ f*8o: see Table 21 



Department /task 
teaching: 



subsidiary services 
(e.g. libraries ) : . . 



N 


range 


fo 


f 2SO 


f 28 O 


92 


0-100% 


4 


48 


7 


90 


0- 60% 


3 


5 


0 


91 


0- 70% 


31 


2 


0 


91 


0- 50% 


31 


1 


0 


90 


0- 15% 


46 


0 


0 


90 


0- 40% 


38 


0 


0 


92 


0- 49% 


26 


0 


0 


95 


0- 50% 


65 


1 


0 



median for teaching: between 48 and 50%; 
median for administration: 17%; 
median otherwise: less than 5% 



For most institutions the main expenditure is on teaching (beside 
expenses for administration). More than one half of the 
institutions (48 from 92) spend 50% or more of their money on 
teaching and 7 from 92 institutions spend 80% or more on this 
task. 

The fact that few private, ^proprietary' correspondence schools 
have answered is evident from the minimal svuns invested in 
advertising/ information. 



Table 23: Share of personnel costs 

in th9 overall costs (L7): 


N«96; (26 x *'no answ€ 
range: 0% to 90%. 
first quartile ^ 40% ^ 
median % 64%; 
third quartile ^ 75% 


5r"); 



PersonneJ. costs are the largest budget item for the majority of 
the institutions: 67 of the institutions spend 50% of their budget 
or more on personnel costs ; 20 of the institutions spend 80% or 
more on personnel. 
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[Table 24: Proportions 


of personnel costs 


for 


the 


departments ( L8 ) : 


to, fiso/ fisos see 


Table '?! 












Department/ task 


N 




Med 


fo 


X%50 






80 




40 


6 


0 D 








0-100% 


18 


3 


8 




- counselling/tutoring 




0- 75% 


4 


25 


2 


0 1 


- subsidiary services: 


81 


0- 60% 


3 


34 


2 


0 


— research/ evaluations 


: 78 


0- 41% 


1 
i. 




0 


0 1 


- exams /certificates : . 




0- 80% 


0 


42 


1 


1 1 






0- 40% 


0 


42 


0 


0 






0- 44% 


0 


62 


0 


^ 1 



The greatest proportion of personnel costs concern 'teaching' and 
'administration'. It is unfortunate that respondents may have 
included the costs for commenting on students' papers and 
assignments either under 'teaching' or under 'counselling/ 
tutoring'. Thus the actual figure for the latter may actually be - 
and probably is - considerably higher than appears from this 
table. 



3,2,8 Co-operation with others 



Table 25: Percentages of services rendered by the institution 
(L9): 



fi.oo% : number of institutions rendering a particular service 
without help from other institutions 

service N fxoo% 

- intellectual production of teaching material: ... Ill 69 

- technical production of teaching material: 110 65 

- counselling/ tutoring: 112 7 8 

- exams/certificates: 112 70 

- evaluation of the teaching materials: Ill 78 

- 'pure' research: Ill 52 

- subsidiary services (e.g. computer center): 112 54 

- advertising/ information ; 112 63 

- administration: 113 82 

- distribution: 113 81 
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The comparatively low number of institutions producing teaching 
material on their own (see Table 25) indicate that several 
institutions obtain their material to some degree from others 
(e*g* from their ^mother' -institutions ) . 

Several institutions employ not only their own staff members, but 
have in addition external teachers as well as external 
counsellors/tutors (primarily employed by other institutions) - 
see Table 26 ♦ 

Apparently the majority of staff members are engaged in teaching 
(cf. Table 24 above; the greatest proportion of personnel costs is 
consumed by ^teaching' ) . 



Table 26: Number of staff members employed in 


the different 


departments ( L2 1 ) : 














N : number of respondents 














Med : median for all respondents 












Med»o: median for institutions 


responding with 


a number greater 


than zero 














fo : number of institutions 


responding with 


It 


zero 






f»o : number of institutions 


responding with 


a 


value 


greater 


than zero 












a) internal: personnel employed by the 


institution 


itself 


N 


range 


Med Med 




fo 






0-2,000 


16 


27 




20 


86 


- counselling/tutoring:... 97 


0-2,150 


3 


5 




27 


70 




0 - 306 


0 


3 




57 


43 




0 - 306 


0 


3 




49 


49 




0 - 306 


0 


6 




57 


46 


- technical services (printing, 
















0 - 300 


3 


5 




33 


70 




0 - 559 


6 


7 




6 


96 


- publicity/information: .. 100 


0 - 300 


1. 


2 




46 


54 




0 - 300 


0 


19 




91 


15 


b) external: personnel employed by other institutions 




N 


range 


Med: 




fo 


f 


»o 




0-4,000 


50 




52 




51 


- counselling/tutoring: ... 103 


0 - 900 


18 




78 




25 




0 - 300 


12 




85 




18 




0 - 134 


6 




91 




10 


- subsidiary services : .... 103 


0 - 134 


14 




96 




7 


- technical services (printing, 














distribution etc.): 103 


0 - 111 


6 




85 




18 




0 - 111 


3 




91 




13 


- publicity/information: . . 103 


0 - 111 


20 




96 




7 




0-88 


30 




100 
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Many institutions {81 out of 106) report cooperation with other 
institutions, with ^traditional educational institutions' and 
other ^distance-education institutions' being mentioned most 
frequently (Table 27 )• 



Table 27: Co-operation with other institutions/type of 
co-operating institution (L26): 

"yes": 104; "no": 15; "no answer": 3 

f : frequency of mentioning 

type of co-operating institution f 

— traditional educational institutions: 81 

— distance-teaching institutions: 74 

— radio or television stations: 44 

— publishing houses: 34 

— administrations: 20 

— advertising agencies: 20 

— others : 19 

(16 X "no answer" for the question as a whole) 



The fields of more intensive co-operation seem to be 
'examinations', 'production of teaching material', and 'use of 
services and buildings' (see Table 28). 



Table 28: Intensity of co-operation with other institutions (L27); I 



5-point response scale from "0: no intensity" to "4: very much" 
N : number of respondents 

f3H.4 : frequency of "3: much" + "4: very much" 



f3-.4 N 

intellectual production of teach: ng material: .... 27 98 

technical production of teaching material: 26 97 

use of media: 19 gg 

study centres: 21 98 

subsidiary services: , 14 98 

examinations: 33 99 

certificates: 22 96 

research/ evaluation : 13 98 

distribution: 16 98 

use of staff : 10 98 

administration: 9 98 

use of services/buildings: 24 98 

publicity: 18 98 

others : 5 gg 
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3.2.9 Importance of compater for the various departments 

The computer is used primarily for administration and for 
(technical and subsidiary) services (Table 29). 



Table 29: Importance of computer for the different departments 
(L22): 



5-point response scale from 
N : number of respondents 
fo : frequency of answers • 
fa^*: frequency of answers ' 



"0: no importance" to "4: very much" 

0: no importance" 
3: much" + ' 



- technical services: 



- others 



very 


much" 




N 


fo 


f 3-^4 




43 


19 




61 


11 




44 


26 




45 


25 




53 


32 




51 


36 




53 


26 




27 


52 




101 


4 



3.2.10 Success, non-starter and drop-out rates 



"Success", "drop-out", and "non-starter" can be defined in 
different ways. 

Although we had tried to give short definitions in the 
questionnaire many responses on the questions in the 
questionnaires show that these terms are used in very different 
ways . 

Accordingly the ^non-response rates' for these questions are 
rather high (This is especially true for the questions referring 
to success, drop-out, and non-starter rates differentiated by 
subject areas) . 

Therefore the rates for the courses with the highest number of 
enrolments (Table 30) may be taken only as rough indicators of 
success and failure. 
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Table 30: Success rate and non-starter rate (average percentage 
for the three courses with the highest number of 
enrolments ) (Q53/54 ) 

- Success rate (Q53) 

range; from 1% to 100% 

median : «67% 

first quartile: . . . . «50% 
third quartile; . . . .«84% 

52 institutions report success rates equal or higher than 80%; 
N=139; 58 x "no answer; 

- Non-starter rate (Q54): 

range; from 0% to 70% 

median; ^=10% 

first quartile; .... » 5% 
third quartile: . . . .^27% 

48 institutions report non-starter rates equal or higher than 
20%; 

N-121; 76 X "no answer; 



The data to Q52 can hardly be interpreted; the 'non response'- 
rates are high (f^o «.,^«w«>r: from 109 to 186). Therefore here only 
some tendencies (see Table 31). 




3*2: General characteristics of the institutions/" 32 



Table 31: Success rate, drop out rate, and non-starter rate per 
subject area (Q52) 



a) . ^cess rate for curriculum ; number of institutions reporting 
success rates of "80% or higher" (f*8o) 

b) drop out rate for curriculum ; number of institutions reporting 
drop out rate of "50% or higher" (f*5o) 

c) non-starter rate for a course: number of institutions reporting 
non-starter rate of "20% or higher" (f*2o) 



Subject areas; 

(1) education, the humanities, music and the arts, 
languages 

(2) social sciences and law 

( 3 ) economics 

(4) agricultural sciences, agricultural and 
sylvicultural professions 

(5) mathematics, sciences and engineering 

(6) medicine and medical jobs and professions 

(7) others 



subject 



area: 




f a:8 O 


Nb 


f %50 


No 


f a:20 




61 


30 


62 


9 


88 


9 


(2) social sciences.. 


27 


6 


35 


7 


53 


7 




21 


4 


33 


8 


50 


6 




8 


1 


10 


2 


17 


3 


(5) math. , science ... 


33 


6 


40 


12 


57 


7 




11 


9 


11 


0 


15 


0 



It is hardly surprising to see that the 'success rates' are 
highest for 'education' and the 'drop-out rates' highest for 
'mathematics and science'. 
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3.2.11 Summary of section 3.2 



- The majority of the institutions in the sample were founded 
after World War II (Table 4) 

- About one half of the institutions are state owned (Table 5) 

- The most frequently mentioned aims of the institutions are: 

— opening of study opportunities to new target groups 
offer of further training opportunities; 

on the other hand, 'profit' is relatively seldom mentioned 
(Table 6) 

- The majority of the institutions offer courses on level III 
(university study and/or further professional training). 
About one third of the institutions are universities (Tables 7 
and 9) 

- Frequently mentioned subject areas for which courses are offered 
are 

— education 

— economics 

relatively seldom offered are courses for: 

— agriculture 

— medicine 
(Tables 8 and 10) 

- The most frequently used teaching mode is 'distance teaching 
with a few face-to-face elements (Table 12) 

- About one half of the institutions have study centres which 
rather have the function of support to distance learning than 
that of compulsory (face-to-face) teaching (Table 13) 

- The institutions differ in the degree of flexibility or in the 
number of options offered to the learners. Most frequently 
mentioned options are; 

— when to start studies 

— when to use counselling services (Table 14). 
Some institutions offer options with regard to the curricula, 
too (e.g.: fixed curriculum, but it contains alternatives) 
(Table 15) 

There are great differences between the institutions with regard 
to: 

— the number of learners (Table 16) 

— the number of courses offered (Table 19) 

— the niomber of staff members (Table 26) 

The courses with the greatest numbers of enrolments are those in 
"education" (Tables 17 and 18) 

The annual budget (in US $) varies considerably; the median is 
between 500,000 and one million (Table 20) 
The most important sources of financial means are: 

— learners ' fees 

— state subsidies 
(laole 21) 

The greatest proportion of the financial means is spent on 
teaching, followed by administration (Table 22) 
Personnel costs are a very important part of the expenses 
(Median: 64% of the overall costs) - (Table 23). And the 
greatest part of these costs concerns teaching (Table 24) 
Many institutions obtain their teaching material from other 
institutions (at least to some extent - Table 25) and/or employ 
personnel from other institutions for teaching and/or 
counselling (Table 26) 



3.2.11 Summary of section 3.2/-34- 



Partners of co-operation frequently mentioned are traditional 
educational institutions and other distance-education 
institutions (Table 27) 

The differences between the institutions with regard to the 
success, drop-out, and non-starter rates are great (Tables 30 
and 31) . ^ v 
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3,3 Learning and teaching^) 
(M* Weingartz ) 

Distance education takes place between the contradictory poles of 
independence and control: 

• How much control is necessary for the learner to be able to 
manage? 

• How little control is feasible in order to guarantee attention 
to the individual learner? 

The .olution of this problem is not only a matter of quantity - we 
must also ask ourselves what measures contribute to fostering 
independence in a meaningful way. 

The results of our first study (cf. Biickmann, N., Holraberg, B.^ 
Lehner^ H. & Weingartz, Steuerung and SelbstMndigkeit im 

Fernstudlumj Hagen, July 1985) show that a considerable majority 
of distance education institutions claim to foster independence. 
Concepts of independent learning and acting differ widely, 
however. These differences refer above all to the degree of 
individualisation^) . 

As far as study activities are concerned, three areas of 
independence may be distinguished: 
study goals 

- carrying out learning tasks and organising study 

- evaluation of study results 



3.3.1 Study goals 

A student studying to pass a specific exam or obtain a specific 
degree, will, as a rule, have little opportunity to choose study 
contents autonomously. There are a few distance education 
institutions, however, who even in such cases offer a large degree 
of freedom and independence (Q22). See Table 15 under 3.2.5 above. 

Almost 50 per cent of the institutions in our sample offer 
students at least the possibility of choosing between alternatives 
v/hen deciding upon study contents. 

Roughly 20 per cent of all institutions let students choose freely 
within a set framework. 

There is also a small group of distance-education institutions 
which take the individual student's needs and practical problems 



1) In this section only the shorter questionnaire is referred to. 
"Q" thus refers to its question numbers. 

2) A high degree of individualisation can be said to exist in 
cases where students can, after prior consultation with their 
teachers or tutors, develop a curriculum adapted completely to 
their wishes and needs. This is then provisionally laid down in 
writing, but can be revised during the course of study. The 
tutor here rather plays the role of a friend and specialist 
giving advice, pointing out possible problems and making 
suggestions. Steuerung and SelbstSndigkeit, p. 7 
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as a starting point , i.e. the learner can, with the help of a 
tutor/ elaborate his/her own study plan or develop an 
individualised learning contract (Q23). 



Table 32: Opportunity for learners to design (with the help of 
tutors) study pr ogrammes tailored to their individual 
needs, for which there does not exist any pre-^oroduced 
study material (Q23): 



"yes": 30 

"no" : 158 

"no answer" : 9 

Use of learning contracts ; 

"yes" : 12 

"no" : 24 

"no answer"/ 

"not applicable": 9 



3 .3 .2 Carrying out learning tasks and organising study 

This area undoubtedly contains the greatest range of variation 
between the two poles of independence and control. 

The answers to the question concerned with the main educational 
aim of the distance education offered by an institution does not 
giye any information about any predominant tendencies amongst the 
institutions (Q24 ) . 



Table 33: Main aim of distance study (Q24): 

"yes" "no" 

- imparting factual knowledge 112 78 

- promoting the development of the ability to 
apply knowledge to: 

- tasks of limited scope 88 102 

- complex problems 102 88 

- encouraging students to make investigations 
of their own 71 119 

(7 X no answer" ) 




If the answers 'imparting factual knowledge' and 'promoting the 
development of the ability to apply knowledge to tasks of limited 
scope' are classified as 'low individualisation' and 'promoting 
the development of the ability to apply knowledge to complex 
problems' and 'encouraging students to make investigations of 
their own' as 'high individualxsation' , the institutions can be 
divided into two groups of roughly the same size. 
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More or less the same applies to question 25. The answers given 
are evenly distributed over the whole range of possibilities. 
About two thirds of the institutions can be considered 'hardly 
individualising' (their examination requirements pux. 'much' or 'a 
lot of stress on the 'reproduction of facts' and "the application 
of knowledge to tasks of limited scope'). The institutions 
stressing individualisation to a high degree are in a minority. 



xcuxc; j*t • CiAutsuu wnxcn examxnacxons stress 


different 


requirements (Q25) : 




5-point response scale from "0: no stress" to 


"4: a lot". 


fa-*--*: frequency of responses "3: much" + "4: a 


lot" 




f 3*4 N 


the reproduction of facts 


..87 185 


- the application of knowledge to: 






..86 182 




..89 181 



One way of fostering independence is working independently on not 
too limited tasks. Self-checking questions might serve as 
incitements to this kind of work. 
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Table 35: Purpose of self-checking exercises (Q26): 



N: Number of responding institutions 



to support retention of facts 132 163 

to make learners practise acquired knowledge . 142 164 

to make learners question claims and basic 

assumptions put forward in study material 69 160 



The tendency mentioned for question 25 is even stronger in the 
answers to this question. Only a bare third of the institutions 
use self-checking questions in order to incite the learner to make 
use of his/her own knowledge and develop solutions or attitudes 
independently . 

The encouragement of study groups or self-help groups is another 
possibility. 



Table 36: Encouraging of study groups (Q27) 



"yes": 124 / "no": 64 / "no answer": 9 



At first sight a surprisingly large number of institutions/ almost 
two thirds, answered this question in the affirmative. Measures of 
this type might, however, be counteracted by deficits in the study 
material, the structure of which may limit independent learning or 
demotivate the learner. 
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3.3.3 Evaluation of study results 

Study results are generally evaluated by the institution, without 
any participation by learners. A fixed curriculum usually also 
means fixed assessment standards, leaving hardly any possibility 
of taking into account learners' self-assessment. This differs 
only for those institutions that make it a task for the learner to 
draw up his/her own study plan. Those institutions also see one of 
their educational objectives in the learner's ability 
to assess her/his own performance. 



Table 37: Participation in assessment of learners' perfomance 
(Q28): 



assessment bv: 


"ves" 


"no" 


N 




167 


22 


189 




28 


161 


189 


- the institution together with 










36 


153 


189 


- the learner; the institution then 


assesses 




12 


177 


189 




3.3.4 Construction of composite scores 

In addition to the analysis of answers to individual questions we 
tried to assess 'independence' by means of composite scores 
allowing the definition of differing degrees of independence. 
Following our assumption that distance education takes place 
between the two poles of independence and control this score of 
independence ought to state the extent of encouragement of 
autonomous actions. Control is as indispensable in distance 
education as in face-to-face education, as the student generally 
does not know from the start what knowledge and skills he/she 
needs to acquire in order to satisfy the requirements of a 
particular profession she/he is interested in. 
The learner therefore needs guidance or control coming from the 
institution. In distance education we can distinguish between 
controlling measures that do not, or only to a small extent, 
consider the individucil learner and above all guide learners 
towards a prescribed aim, and those controlling measures that 
include the skills, level of knowledge, needs and wishes 
particular to the individual learner. 

In the following passages we therefore distinguish between a more 
goal-oriented type of control and a more individualised type. We 
assume 'chat goal-oriented control frequently makes use of measures 
that may lead to a restriction of the individual's work and 
therefore in the long run do not really suppport independence but 
actually could counteract it. 
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3.3.4.1 Score of independence 

The score is to indicate the extent of support enablina the 
learner to keep up self-directed study activities ie activities 
that do not depend on the presence of a teachej activities 

The score is to rise 

- if the institution enables the learner to be flexible with 
regard to Eacina, and if at least two of the foUowing 
flexibility components apply: 

- when to start a course/the studies (Q9a) 

- when to take exams /final exams (Q9b) 

- when to order study material (Q9c) 

- when to submit assignments (Q9d) 

" conJ^ni-=''^^^^U^^x'' students options regarding study 
' °ne of the following components applilsl 

- the curriculum is fixed, but contains alternatives (Q22b) 

■ (5220^''^'''' ^""^^^^ ^ ^^'"^ 

- study programmes may be tailored to individual needs (Q23al 

- learning contracts (Q23b) (U^Jaj 

" institution mainly aims at promoting the development of 

problem-solving abilities, and if Sne of the following^ . 
components applies^); "-i-"y 

' lil institution mainly aims at promoting the development of 
the ability to apply knowledge to complex problems or at 
tS24br?iS' Q^J^r'^ ^° '"^'^ investigations of their own 

" ?rcompL'rprog?:r!S25\2r ^PP^^-^^- knowledge 

- if the institution uses self-checking exercises in order to make 
s™t^"nr^S26ct'""^ "^'^^ — P"°ns put forward 

" considers it necessary constantly to stay in 

Je?t wTii .i^^i^H^^J^'v'^^^ initiative to this, however, is not 
have hoon ^^^^®^ts but the institution contacts those who 
nave not been m touch for a certain time (Q34a). 

- if the assignments for submission are commented on as 
extensively as possible as that kind of feedback is most likely 
to make the learners feel secure and give them hints on their 

n^af i?^?^^^-^''' (Q47a,b,d: comments of half to more than one 
page of the size of this report) . 

' ^5 the institution follows its central idea even in the area of 
leainers°(Q28b,c)!^^^®^ learner performance in cooperation with 



3) It was possible to combine these two characteristics, because 
our first survey already showed a significant correlation there 
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Table 


38: Item 


-intercorrelations for the score of independence* > 






*; 


p< .01 


+: p< .05 




R9b 


nil /IT 


RZ't/ ZD KzbC 


K j4 K4 / RzoDC 


R2223 


-0.07 








t\ ^ ^ ^ J 


-0 09 


0. 16* 






R26c 


-0.02 


0.08 


n 0 o * 




R34 


0.22* 


0.05 


0.07 0.09 




R47 


0.08 


0.06 


0.14+ 0.19* 


0.13+ 


R28bc 


-0.00 


-0.03 


0.14+ 0.12 


0.08 0.01 


item- 


scale correlation 


(corrected for 


the part-whole effect): 




0.04 


0.08 


0.26 0.27 


0.10 0.22 0.21 


Scale 


reliability 


coefficient 


of reproducibility 


(Proctor 1971) 




(Guttman) 






rt.t.= .4l 






.71 



The table above shows that the following correlations are 
significant: 

- If there is a tendency for the institution mainly to stress as 
the aim of its teaching the development of learners 'problem- 
solving abilities (R24/25), 

then the institution provides a rather open curriculum 
(choice between alternatives or free choice within a set 
framework or even drawing up individual study 
plans) (R22/23); 

then students are more frequently encouraged to question 
claims and basic assumptions put forward in study material 
(R26c); 

then assignments for submission are commented on as 
extensively as possible (R47a,b,d); 

then learner perfoimance is more frequently assessed in 
cooperation with students or the learner assesses his/her 
own performance and the institution then assesses the 
learner's assessment (R28b/C). 

- If distance education institutions attach importance to the 
learners' questioning claims and basic assumptions put forward 
in study material (R26c) or keep in constant touch with the 
students instead oi leaving the initiative to them (R34)/ then 
there is also extensive commenting of assignments (R47a,b,d)- 

- If the institution itself contacts students who have not been, in 
touch for some time (R34), there is the additional tendency of 
flex.fbility in pacing (R9b). Flexibility in pacing does not play 
an important role otherwise; there even is a slightly negative 
correlation with a few items. The reason for this might be that 
with highly individualised forms of education, pacing is of 



4) Each item is transformed in such a way that it is scored one 
("1") if it points in the direction of 'independence' and is 
scored zero ("0") otherwise. This implies that 'non response' 
is scored as zero. The scores of individualised control viz. 
Q goal-oriented control - see below - are constructed in the same 
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considerable importance. It is one of the factors in learnino 
contracts, for instance. 

3.3.4.2 Score of individualised control 

The score is to indicate the extent of that control that takes 
into account and activates learners' previous knowledge, 
individual characteristics and aims. 

The score is to rise 

- if the function of study centres is to support distance 
learning, but participation is not compulsory (Q17b) 

- if two-way communication is a constitutive element of the 
services offered by the institution , i.e. there is regular 
mediated or direct personal contact between tutor and student 
VIZ. there is a response to learners' queries, requests etc. 
(Q29cl,2,e) 

- if the institution offers individualised tutoring and 
counselling services (Q30a2,b,c2,d = Q30T) 

- if learners are assigned a personal tutor/counsellor (Q36) 

- if assignments for submission are commented on (Q46) 

- if tutors' comments refer to students' individual achievements 
(Q48) 

- if the person who corrects and comments on students' assignments 
IS also responsible for counselling (Q49a,b,c) 



Table 39: Item-intercorrelations of the score of individualised 
control 



*: P^.Ol +: p<.05 



Q29cl,2,e 


Q17b 


Q29cl, 


2,e Q30T 


Q36 


.06 




Q30 T 


.05 


.18+ 






Q36 


-.06 


.04 


.08 




Q46 


.03 


.08 


.12+ 


-.03 


Q48 


.04 


.18* 


.14+ 


-.08 


Q49a,b,c 


.03 


.26* 


.16+ 


.07 



Q46 Q48 Q49a,b,c 



.41* 

.50* .31' 



item-scale correlation (corrected for the part-whole effect) 
^^■■^ -05 .25 .22 .01 .40 .33 0.46 



Scale reliability coefficient of reproducibility 

(Proctor 1971) (Guttman) 

rtt=.76 ro=.89 



The table above shows that the following correlations are 
significant: 

If two-way communication is a constitutive element of the services 
offered by the institution (Q29cl,2,e), 

- then there are also individualised tutoring and counselling 
services (Q30 T); 
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- then tutors' coimnents also refer to individual achievements 
(Q48); 

- then the person who corrects and comments upon students' 
assignments is also responsible for counselling (Q49a,b,c). 



If assignments for submission are commented upon (Q46) 

- then tutors' comments also refer to individual achievements 
(Q48); 

- then the person who corrects and comments upon students' 
assignments is also responsible for counselling (Q49a,b,c). 

And: if tutors' comments also refer to individual achievements 
(Q48) then the person who corrects and comments upon students' 
assignments is also responsible for counselling (Q49a,b,c). 



Question 17 (the use of study centres to support distance 
teaching, no compulsory attendance) does not correlate with the 
other items/conditions. As we already assumed in our analysis in 
the interim report (Biickmann et al 1985, p. 46), counselling in 
study centres is likely to be of little importance for 
institutions using highly individualised forms of control. 

Question 36 (a student being assigned one single tutor) is of no 
significance. This result contradicts that obtained in our first 
survey. Possibly the reason could be that this item was answered 
in the affirmative by only 10 per cent of the institutions in our 
Scunple . 



3.3.4.3 Score of goal-oriented control 

The score is to indicate the extent of control exerted on the 
learner in order for him to reach a fixed aim (degree, diploma) as 
effectively as possible. 

The score is to rise, 

- if the institution encourages study groups (Q27) 

- if the institution offers particular standardised procedures for 
tutoring and counselling (Q30al) 

- if the institution encourages the learners to approach their 
tutors when they feel they need assistance (Q33) 

- if the institution employs measures to reduce the non-starter 
and drop-out rates (Q35a,c = Q35T) 

- if the institution uses continuous assessment (Q38) 

- if the assignments for submission mainly serve the purpose of 
assessing learner performance (Q39) 
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control 




Q27 


Q30al 


Q33 


Q30al 


-.08 




Q33 


.13+ 


. 18* 




Q35 T 


-.15+ 


.33* 


.02 


Q38 


-.05 


.13 


.14 


Q39 


-.07 


.01 


.04 



Q35T Q38 Q39 



.08 
-.05 



,07 



item-scale correlation (corrected for the part -whole effect); 
ri-t: -.11 .24 .22 .15 -.02 .09 



Scale reliability 
(Proctor 1971) 

rt:t:=.51 



coefficient of reproducibility 
( Guttman ) 

rp=.81 



The table shows some interesting correlations between the 
following items: 

If the institution offers standardised procedures for tutoring and 
counselling (Q30al), 

- these consist, among other things, in encouraging learners to 
approach their tutors when they feel they are in need of 
assistance (Q33); 

- it will more frequently send students preproduced (standardised) 
reminders/letters (or tapes) of encouragement as a means to 
reducing non-starter and drop-out rates (Q35T); 

- it will use continuous assessment (Q38) 



These items partly also correlate among each other (Q33 - Q38): 

If the learners are encouraged to approach their tutors (Q33), 
this also entails stronger encouragement of study groups (Q27) . 
Tne encouragement of study groups is of no other significance for 
this score. In our first survey our assumption that it is exactly 
in a situation of stronger control that study groups are 
encouraged (possibly to counterbalance that control) had been 
broadly confirmed. 

Individualised courses of study may as such produce higher 
motivation for communication since everyone works on different 
problems and solutions. The importance generally attributed to 
study groups is such that there can be no general confirmation of 
our assumption. 
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3.3.5 Distribution of Scores 

The score for independence extends over seven levels, which 
results, for the 197 institutions in the sample, in the following 
distribution: 



Table 41a: Distribution of the score for independence 

range: 0-7 
first quartile = 2 
median = 3 

third quartile = 4 



The score for goal-oriented control extends over six levels, 
resulting in the following distribution: 



E 



Table 41b: Distribution of the score for ga^l^oriented control 



range: 0-6 
first quartile = 2.5 
median - 3 

third quartile = 4 



The score for individualised control extends over six levels, 
resulting in the following distribution: 



[ra 



Table 41c: Distribution of score for individualised control 



range: 0-6 
first quartile = 3 
median - 4.5 

third quartile « 5 



3.3.6 Correlations between the scores of independence and control 
and the success rate 



The success of distance education institutions is usually measured 
by the study success of their students. 

In this context, success is measured by the following items: 

- Question 53: How high is, in the three courses with the highest 
number of enrolments last year, the average percentage of those 
finishing a course successfully? 

- Question 54: How high is, in the three courses with the highest 
number of enrolments last year, the average percentage of those 
who after enrolment have submitted no assignment? 
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Table 42: Correlations of scores for independence and of control 
with success 



Scores : indep . 11 goal-or 

control 



success (Q53) 
non-starter rate (Q54) 



.19 



.07 



-.19 I -.01 



Individ, 
control 



.24 
-.08 



The components most crucial to success are apparently a relatively 
high amount of independence and forms of control that are as 
highly individualised as possible. As a consequence these 
components show a negative correlation with the percentage of 
assignments not returned to the institution. Goal-oriented control 
or rather thoje, mostly standardised, measures that are subsumed 
under this heading, does not seem to have any significant effect 
on success . 



3, 3,7 Summary of section 3.3 

Autonomy is an important educational aim not only in distance 
education. It is the situation of the distance-education learner 
that makes the question about independent behaviour a special 
problem. 

On the other hand difficulties in attaining and fostering 
independence are easier to define for distance-education 
institutions because of the typical teaching/learning organisation 
where elements like study material, two-way communication are 
separated in advance. Thus the defining of measures which help to 
foster independence in distance education might be fruitful also 
for other types of educational institutions. 

Apart from analysing answers to individual questions concerning 
study goals, study organisation and evaluation of study results we 
constructed a score defining different degrees of independence. 
Since distance education is not possible without control, 
especially at the beginning of a course, we therefore constructed 
two further scores which distinguish a more goal-oriented type of 
control and a more individualised type of control. 
The question is which measures of cor.trol support independence and 
which measures might counteract it. 

A final correlation of the scores with the success and non-starter 
rate showed that measures of independence are important for the 
study process. The measures of individualised control seem to be 
of similar importance. 



These results raise the question, to what extent, if at all, 
control has any effect on the course somebody's studies take or if 
maybe measures fostering independence might suffice. This question 
will be discussed in a forthcoming publication. 
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(F. Doerfert & C. See^BogehojJj 

With FernUniversitat experiences and results from previous studies 
in mind, we had very specific expectations as to the outcome of 
this survey. These are reflected by our comments on the individual 
tables. 



Table 43: Media used 


for each subject area 










subiect areas: 








(1) 


education, the humanities, 


music and the arts. 




languages 












(2) 


social 


sciences and law 








(3) 


economics 












(4) 


agricultural 


sciences 


, agricultural 


and 






sylvicultural 


professions 








(5) 


mathematics , 


sciences 


and engineering 




(6) 


medicine and 


medical 


jobs 


and professions 




(7) 


others 














Medixun/subiect areas: 














(1)....(2)... 


.(3).... 


(4)... 


.(5).... 


(6).. 


..(7).... 


Sum 




radio 
















42 21 


16 


6 


14 


5 


8 


112 




TV 
















35 25 


16 


4 


18 


6 


8 


112 




written course unit 














130 89 


84 


31 


87 


28 


43 


492 




audio tape 
















97 45 


32 


8 


30 


16 


21 


249 




video tapes 
















55 30 


26 


11 


35 


14 


15 


186 




film 
















28 12 


13 


6 


11 


6 


4 


80 




slides 
















35 12 


11 


11 


22 


8 


5 


104 




electr. data 


processing 












10 4 


7 


2 


16 


4 


7 


50 




PC 
















17 7 


13 


4 


33 


5 


8 


87 




telephone 
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37 


10 


33 


14 


16 


204 




face-to-face 


sessions 














83 53 


46 


12 


53 


16 


19 


282 




laboratories /workshops 












34 16 


12 


7 


46 


12 


7 


134 




others . 
















7 7 


10 


6 


22 


5 


7 


064 




Smn 
















628 360 


323 


118 


420 


139 


ler 


2156 
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Printed material is the main medium in all subject areas as was 
to be expected; rather unexpextedly, however, face-?o!?Ice 
sessions are frequently used in distance education - amazingly 
altogether more frequently than telephone tutoring. ^""^^^"^^^ 

veL?Srof%^S corresponding question (L30) in the longer 
?ables 4?f M questionnaire show a more detailed picture (see 
?ho ?n„! S:^'* ^^^""^ subject areas are more differentiated. 
The long version was answered by 122 institutions (N=122). 

- use of media (L30 W 

85 out of 116 institutions (6 x "no answer") say that they use 
different media for different subject areas 

per "^t'ln'Table " distributed over the media (sum 



Table 44a: Use of media (L30) 

f : frequency of mentioning 
% : percentage related to the over all total of 1736 
nominations 



medium 

written course unit: 

face-to-face sessions: ... • 

audio tapes : 

telephone: 

video: 

laboratories /workshops : . . . 
radio : 

TV: 

slides : 

PC: 

film: 

electronic data processing, 
others : 



f 


% 




446 


25. 


7% 


234 


13. 


5% 


201 


11. 


6% 


150 


8. 


6% 


142 


8. 


2% 


106 


6. 


1% 


101 


5. 


8% 


83 


4. 


8% 


76 


4. 


4% 


73 


4. 


2% 


61 


3. 


5% 


33 


1. 


9% 




1. 


8% 



6 

ERIC 
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media used frequently for each sublect area (in brackets: the 
frequency of mentioning - see also Table 44b) s 

- Education # teacher training; Written course unit 

(48), face-to-face sessions (32), audio tape (25) and video 
(23); sum of nominations for this subject area; 239 

- Humanities, music and the arts ; Written course unit (45), 
face-to-face sessions (25), audio tape (23) and radio (16); 

sum of nominations for this subject area: 176 

- Languages > linguistics ; Written course unit (56), 

audio tape (39), face-to-face sessions (29), video (14)/^ 
Radio and telephone (13 each); sum of nominations for this 
uubject area: 191 
« Law and legal professions ; Written course unit (25), 

face-to-face sessions (15), audio tape (11), radio (5) and 
telephone (6); sum of nominations for this subject area; 82 

- Economics ; Written course unit (55), face-to-face sessions 
(27) , audio tape (20) and telephone (19); sum of nominations 
for this subject area; 190 

- Agriculture ; Written course unit (16), slides and telephone 
(7 each), video and face-to-face sessions (6 each); sum of 
nominations for this subject area; 67 

- Engineering and technical professions ; Written course unit 
(32), laboratories /workshcns (23)/ face-to-face sessions (19) 
and PC (17); sum of nominations for this subject area; 164 

- Sciences ; Written course unit (37), face-to-face sessions 
(20), laboratories /workshops (18), video and audio tapes 
(12 each); sum of nominations for this subject area; 145^ 

- Mathematics; Written course unit (41), f ace-to-»f ace sessions 
(16)/ PC (10), audio tapes (10); sum of nominations for 
this subject area; 118 

- Social sciences ; Written course unit (49)/ face-to-face 
sessions (27)/ audio tapes (23), telephone (22), video (18) 
and TV (15); sum of nominations for this subject area: 193 

- Medicine and medical jobs ; Written course unit (19), audio 
tapes (12), face-to- face sessions (11)/ and video (9); 
sum of nominations for this subject areai 88 



Table 44b; Media used for each subject area (L30) 



1 
2 
3 
4 
5 
6 
7 
8 
9 
10 
11 
12 



Subject areas : 

education, teacher training 

humanities, music and arts 

languages , linguistics 

law and legal professions 

economics 

agriculture, sylviculture/ forestry 

engineering and technical professions 

sciences 

mathematics 

social sciences 

medicine and medical jobs 

others 



{Continued) 
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3.4 Teaching Media/ "50" 



Table 44b; Media used for eac h subject area (L30) - Continued 

Medium/subjects areas: 

(5) (6) (7) (8) (9) (10) (11) (12) Sunune 

Radio 

1^ 05 09 05 03 07 06 12 04 04 101 

TV 

17 08 06 04 08 03 06 06 05 15 03 02 83 
Written course unit 

45 56 25 55 16 32 37 41 49 19 23 446 

Audio tapes 

25 23 39 11 20 



10 12 10 23 12 12 201 

9 6 142 



Video 

23 15 14 03 14 06 16 12 06 18 
Film 

11 09 06 03 08 02 06 03 03 05 03 02 
Slides 

1^ 09 03 02 06 07 09 09 04 06 04 04 

Electronic data processing (mainframe) 

5 00 01 01 03 01 08 02 03 01 02 06 

PC 

07 02 02 03 08 04 17 06 10 04 04 06 
Telephone 

22 16 13 06 19 07 12 11 07 22 08 07 150 

Face-to-face sessions 

^2 25 29- 15 27 06 19 20 16 27 11 07 234 

Laboratories/workshops 

17 06 07 03 08 03 23 18 04 08 06 03 106 



63 
76 
33 
73 
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3.4 Teaching Media/-51' 



Table 45: Particular prescribed combinations of media 



"yes": 58 / "no": 104 / "no answer": 35 



Compulsory media combinations are used much more frequently than 
we had expexted* 



Table 46: Share of the different media in the combinations 



N : number of respondents 
fo : frequency of answers "0" 

Med»o : median for those institutions reporting percentages 
^greater than zero' for a medium 



Medium N range fo Med:, 



radio 124 0- 80 97 5 

TV 123 0- 70 101 3 

written course unit 125 0-100 6 70 

audio tape 123 0- 80 48 5 

video tapes 126 0- 40 73 5 

film 128 0- 10 112 2 

slides 128 0- 5 99 2 

electr^ data processing. 127 0- 20 109 2 

PC 126 0- 40 94 3 

telephone 126 0-100 70 5 

face-to-face sessions ... 125 0- 80 50 10 

laboratories /workshops. .129 0- 47 100 5 

others 129 0- 50 118 4 



Table 47: Course teams (Q13): 



"yes": 145 / "no": 38 / "no answer": 14 



This extensive use of course teams corresponds with our 
expectations . 
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3.4 Teaching Media/''52'' 



Table 48: Participants of course teams (Q14): 



f : frequency of tick-offs 
% ! percentage related to the total number of nominations (N=229) 

^ % 

- subject specialists/students of the subject/ 

media specialists 40 17,5 

- subject specialists/media specialists 79 34*5 

- subject specialists 110 48*0 



It can be regarded as very interesting that quite a munber of 
institutions include students in their course teams; the tasks 
assigned to students in this context will be subjected to further 
study. It is surprising that almost half of all institutions 
exclusively employ subject specialists for course development. It 
will be worth while finding out whether the material produced by 
these institutions shows a deficit as to media aspects. 



Table 49: Internal/external media specialists for 
development (Q15 ) 


course 


- internal/members of staff of the institution 


..66 


- external/outside specialists 




- partly internal/partly external 




(48 X "no answer'* ) 



Many institutions also consult external media specialists for 
course development. 



Table 50: Study centres and their f unction- (Q16/17 ) 



- study centres: ves/no 

"yes": 99 

"no": 94 

"no answer" : . .4 



- function of study centres : 

the study centres have rather the function of support of 
distance learning (voluntary participation) (72 x) than the 
function of compulsory face-to-face teaching (54 x) 
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3.4 Teaching Media/ "53' 



Table 51: Evaluation of study material (Q18): 1 


the evaluation refers to: 






- particular course units . . 

- particular curricula 

- media aspects 


"yes" 
. .139 
79 


"no" 
28. 
88 
90 


(30 X "no answer) 







There is more frequent evaluation of media than we had expected, 




Table 52; Selection of media used (Q19): 

"yes" 

- mostly on the basis of general guidelines 52 

- mostly on the basis of guidelines formulated 
within the institution HI 

- mostly as a matter of intuition 37 

- on the basis of material produced by other 
institutions 44 

- others 44 




"no" 



132 

73 
147 

140 
140 



Table 53: Estimated percentage of expenditure on media - related 
to the overall budget (Q20) 



N=113 respondents (84 x "no answer") 
range: 0-100% 
Median » 12 

fa:so = 13 (number of institutions spending 50% or more 
of their budget on media) 



It is remarkable that some institutions should state that they use 
50% or more of the money at their disposal for media development. 



Table 54: Number of members 


of staff developing media (Q21): 




N=144 respondents 


(53 X "no answer" ) 




range: 0-380 




Median - 4 






up to 10 members: 


104 institutions 




from 11 to 50: 


27 




more than 50: 


13 
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3.4 Teaching Media/ 54 



It was to be expected that in distance education a comparatively 
large number of staff members would be employed in media 
development. 



Summary of section 3.4 

In conclusion v/e may say that most of the results were in 
accordance with our expectations. The medium primarily used in 
distance education is printed material. It is^ however, remarkable 
that face-to-face sessions should come second in frequency^ This 
might point to shortcomings in purely mediated distance teaching 
which are essentially compensated for by face-to-face sessions. 

We would also like to mention an investigation currently being 
carried out in the media domain. Media specialists at different 
institutions were approached directly in order to obtain detailed 
and extensive information for the analysis of the media 
presentation of distance study material. 



fi3 
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3.5 Two-way comm jiication: tutoring, counselling and assessment 

(R. Schuemer) 



The following two sections describe frequencies of responses on 
questions with regard to 'tutoring and counselling' (3.5.1) and to 
"assessment of learners' performance' (3.5.2) . Section 3.5.3 
presents a concept attempting to integrate some of the variables; 
their intercorrelations and the relationships between them and the 
success rate are described. 



3.5.1 Tutoring and counselling 

90% of the institutions (176 of 194; 3 x "no answer") offer a 
counselling and tutoring service (Q30). It includes telephone 
service and written correspondence at the great majority of 
institutions (151 x and 145 x, resp.; see Table 55); but also 
opportunities for counselling by face-to-face contacts are offered 
by more than an half of the institutions (116 x); tutoring/ 
counselling by audio tape is offered less frequently (60 x). 

The written correspondence for tutoring contains individualised ' 
letters (127); standardised letters or pre-programmed text modules 
are used less frequently (71 and 32 x , resp.). 

The tutoring/counselling service can be contacted not only on 
weekdays during the usual office hours but also beyond these hours 
in some institutions (see Table 56a/b). 



Table 55: Media used in two-way communication/counselling and 
tutoring service (Q30): 

"yes" "no" N 

- number of institutions with counselling 



and tutoring services: 176 18 194 



Media: "yes" "no" N 

- written correspondence: 145 26 171 

- standardised letters: 71 74 145 

- individualised letters: 127 18 145 

- use of text-modules: 32 113 145 

- telephone: 151 28 179 

- audio tapos: 60 91 151 

- direct (lace-to-face) contacts: 116 32 148 



34 institutions use each of the 4 media (written correspondence, 
telephone, audio tapes, and face-to-face contacts), 71 use 3 
media, and 53 use 2 of them. 
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3,5 Two-way communication: tutoring, counselling and assessment/ "56- 



Table 56a: Times/hours for contacting the counselling services 
by telephone (Q31): 



"yes" N 

- on workdays during usual office hours: 147 168 

- on workdays after usual office hours: 42 168 

- on week-ends: 33 i68 

- other: 9 ig? 



Table 56b: Times/hours for contacting the counselling services 
directly (e.g. in study centres) (Q31): 



"ves" N 

- on workdays during usual office hours: 72 168 

- on workdays after usual office hours: 19 168 

- on week-ends: 30 168 

- other: 4 167 



The estimations of the proportion of learners using the services 
(Q32; see Table 57) vary to a high degree; the overall proportion 
is rather low (median between 35 and 40%). 

The corresponding data - differentiated according to subject area 
- cannot be interpreted because of high occurrences of "no answer" 

(fno An«w*3Er ^ 160) . 



Table 57: Percentages of learners using the counselling 
services (Q32 ) : 



N=117 respondents (80 x "no answer") 
median: between 35% and 40% 

first quartil: between 10% and 15% 
third quartil: between 75% und 80% 
range: from 0 to 100%; 



Almost all institutions (192 out of 197) consider two-way 
communication a constitutive element of their services (Q29), with 
correction of and commenting on assignments playing an important 
part (see Table 58). 



f>5 



3*5 Two-way communication: tutoring, counselling and assessment/" 57 



Table 58: Elements of two-way communication (Q29): 



f : frequency of mentioning 

% : percentage related to N=934 nominations 



correction of assignments 172 



% 



(18 .4%) 
(18.7%) 



commenting of assignments 175 

regular contacts between tutor and learner: mediated 

contact 84 ^ 9,0%) 

regular contacts between tutor and learner: direct 

face-to-face contact 98 

organised face-to-face sessions 114 

answering of learners 'queries , requests etc 169 

contacts with learners initiated by the institution, 99 
other 23 




(10.5%) 
(12.2%) 
(18.1%) 
(10.6%) 
( 2.5%) 

The great majority of the institutions uses several of these 
elements; the number of elements per institution varies between 0 
and 8; the distribi:tion: 
0-2 elements: 17 x ( 8.6 % for N=197) 
3-4 elements: 69 x (35.0 %) 
5-6 elements: 75 x (38.1 
7-8 elements: 36 x (18.3 %) 

The majority of the institutions take some measures to reduce the 
non-starter and drop-out rates. The 'mailing of standardised 
letters of encouragement' is the most frequently taken measure 
(see Table 58). Audio tapes are very seldom used for this purpose. 



Table 58: Measures to reduce the non-starter and drop-out rates 
(Q35): 



_J, 

mailing of pre-produced (standardised) letters of 

encouragement or reminders 112 33 

mailing of individualised letters 93 28 

mailing of pre-produced (standardised) encouraging 

audio tapes 5 1 

mailing of individualised audio tapes 2 1 

phone calls to learners 78 23 

visit to learners by people appointed by the 

institution 27 8 

other 13 3 

(% for N=330 nominations; 23 x "no answer" for the question 
as a whole) 



% 



9 

,2 

5 
0 
6 

2 
9 



The number of measures taken by an institution varies between 0 
and 7. The distribution of the number of measures per institution: 

0 measure: 36 x (18.3 % for N=197) 

1 measure: 61 x (31.0 %) 

2 measures: 54 x (27.4 %) 

3 measures: 28 x (14.2 %) 

4 or more : 18 x 9.1 %). 



3*5 Two*-way communication: tutoring, counselling and assessment/ 58 



The tutori-7/counselling service of the majority of institutions 
is organisei in such a way that the learner has to contact 
different tutors if she/he is enrolled for several courses; only 
at a few institutions a learner seems to be assigned one central 
tutor for all her/his courses (with regard to this problem see 
Rekkedal 1985). 



Table 60: Tutors ' /counsellors ' responsibility if a learner 
has enrolled for several courses (Q36) 



- one tutor assigned to learner: 25 

- different tutors for different courses of a learner: 123 

- partly the same tutor for different courses, 

partly different tutors for different courses: 13 



(N=161; 36 X "no answer") 



In similar ways at many institutions students have to contact 
different tutors/counsellors for different problems. 



Table 61: Responsibility of the tutors/counsellors for different 
problem areas (Q37) 



- common for all problems of a learner: 7 2 

~ different counsellors for different problems: 82 

- partly common, partly different: 6 



(N=160; 37 x "no answer*') 



- EncouraqincT of study groups (027^ ; 

About two thirds of the institutions encourage their learners to 
form study groups or self-help groups (Q27: 124 x "yes", 64 x 
"no"; 9 X "no answer" ) . 



3.5.2 Assessment , exams 

Almost all institutions use some form of continuous assessment 
(Q38: 183 x "yes", 9 x "no", and 5 x "no answer"); ^written 
assignments' and to a lesser degree 'written intermediary exams' 
are mentioned very often. 



3,5 Two-way communication: tutoring^ counselling and assessment/" 59 



Table 62: Type of continuous assessment (Q38) 

no 

yes no answer 

- written assignments: 168 16 13 

- written intermediary exams: 106 77 14 

- oral intermediary exams: 34 149 14 

- others: I9 154 14 



'Assessment of learners' performance' and 'support of learner' are 
equally often mentioned purposes of the assignments. 



Table 63: Main purpose of assignments (Q39): 



- assessment of learner performance (mainly exam function) 52 

- learner support 51 

- partly, partly / both 84 



(N=187; 10 X "no answer") 



The great majority of the institutions require students to take 
final courses examinations (040; 169 x "yes", 25 x "no", and 3 x 
"no answer"). The examinations are in most cases written. 



Table 64: Type of final course examinations for courses (Q40): 



no 

yes no answer 

- written exams: 144 23 30 

- oral exams: 22 145 30 

- written and oral exams: 44 123 30 

- others : 3 7 149 31 



In more than half of the institutions the acquisition of course 
certificates and final marks depend both on the marks obtained in 
the final examination as well as on the marks awarded for 
assignments . 



68 



3,5 Two-way communication: tutoring, counselling and assessment/ '60' 



Table 65: Acquisition of course certificate and final marks 
for courses depending (Q41): 



no 

no answer 



-I . ^ yes 

- exclusively on the number of points obtainsd 
or the marks acquired in assignments: 44 133 20 

- on the marks obtained in the inal 
examj- tion as well as on the marks 

aware :^ for assignments: 119 53 20 

- exclusively on the student's achievements 

in a final examination: 49 128 20 

- other: 17 ^gQ 20 



An investigation by B^^th (1980) has pointed to the importance of 
the regularity and frequency of assignments for the learners' 
success. About two thirds of the institutions responding to Q42 
report having assignments in every course unit. 



Table 66: Number of course units containing assignments (Q42): 
assignments in : f % 



- less than half of all course units: ly 13.0 

- approx. half of all course units:.... 17 11.6 

- more than half of all course units: 16 11.0 

^ every course unit: 94 64! 4 



(% for N=146; 51 "no answer") 



About two thirds of the institutions allow the learners to submit 
their assignments whenever it suits them (Q43: 116 x "yes", 66 x 
"no", and 15 x "no answer"). 

The average turn-around time for correction of assignments (i.e. 
the time lag from the day a student's assignment arrives until it 
is returned with the tutor's corrections and comments: Q44) varies 
considerably between institutions; about one half of the 
institutions needs 10 or more days; a turn-around time of 14 days 
or less is kept by 71% of the institutions. 



ERLC 
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3.5 Two-way communication: tutoring, counselling and assessment/ "61** 



Table 67: Average turn-around time for correction 
of assignments (Q44): 



median: between 9 and 10 days 
first quartile: • • • » 5 
third quartile: . • ^^IS 

mean: 14,4 

range: from 1 to 300 days 



23 X "no answer" 



Predominant types /formats of items (045) are 'short open-ended 
questions', 'short essay on a set subject', and 'multiple choice '- 
items • The type of items preferred depends to some degree on the 
subject area; types used frequently for a certain subject area 
are: 

- for (1) education, the humanities, music and the arts: short 
essay (100 x), short open-ended questions (91 x), and multiple 
choice (70 x) (Total of nominations for this subject area: 459) 

- for (2) social sciences and law: 'short essay' (70 x), 'short 
open-ended questions' (58 x), 'longer paper on a set subject' 
(54 X), and 'multiple choice' (50 x) (Total of nominations for 
this subject area: 305) 

- for (3) economics: 'short open-ended questions' (62 x), 'short 
essay' (56 x), and 'multiple choice' (52 x) (Total of 
nominations for this subject area: 264) 

- for (4) agriculture, : 'short open-ended questions' (21 x), 
'multiple choice' (20 x), and 'short essay' (17 x) (Total of 
nominations for this subject area: 90) 

- for (5) mathematics, sciences, and engineering: 'short open- 
ended questions' (63 x), 'multiple choice' (60 x), 'questions to 
be filled-in on forms prescribed - e.g. numerical answers' (50 
X), and 'short essay' (44 x) (Total of nominations for this 
subject area: 260) 

- for (6) medicine and medical professions: 'multiple choice' 
(19 X), 'short open-ended questions' (18 x;, and 'short essay' 
(15 X) (Total of nominations for this subject area: 85) 
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3.5 Two-way communication: tutoring, counselling and assessment/ ••62" 



Table 68: Types/fo rmats of items/tasks used in the assignments 
for the different subject areas (Q45): 



Subject areas : 

(1) education, the humanities, music and the arts, 
languages 

(2) social sciences and law 

( 3 ) economics 

(4) agricultural sciences, agricultural and 
sylvicultural professions 

(5) mathematics, sciences and engineering 

(6) medicine and medical jobs and professions 

(7) others 



Subject areas ; 
Type of task/item ; 



(1)..(2)..(3)..(4)..(5)..(6)..(7) 



► Sum 



questions 91 


58 


62 


21 


63 


18 


32 


345 


multiple choice 70 


50 


52 


20 


60 


19 


29 


300 


questions to be filled-in 
on forms prescribed (e.g. 
numerical answers) 41 


30 


33 


12 


50 


10 


19 


195 


short essay on 

a set subject 100 


70 


56 


17 


44 


15 


30 


332 


longer paper on 

a set subject 71 


54 


34 


08 


05 


■ 11 


19 


202 


longer paper on a self 
chosen -subject 38 


27 


12 


05 


11 


06 


08 


107 


sample of student's work/ 
workpieces 40 


07 


10 


05 


20 


04 


15 


101 


others 08 


09 


05 


02 


07 


02 


02 


035 


Sum 459 


305 


264 


090 


260 


085 


154 


1617 


(N=184; 13 X "no answer" for 


the question 


as a 


whole ) 







The answers to the longer version of tne questionnaire gives a 
somewhat more detailed picture; there the subjects areas are more 
differentiated (Question Ko. 69 in the long version: L69): Table 
69a contains the frequency of mentioning of each type of item; the 
data - differentiated for each subject area - are in Table 59b. 
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3.5 Two-way coiomunication: tutoring, counselling and assessment/ "63" 



Table 69a: Type/format of items/tasks of assignments (L69): 
f : frequency of mentioning 

% : percentage related to the total of nominations (N=1448) 

type _j % 

short open-ended questions 321 22.2 

multiple choice 26 2 18 •! 

questions to be filled-in on forms prescribed 153 10.6 

short essay on a set subject 308 21 .3 

longer paper on a set subject 209 14.4 

longer paper on a self chosen subject 9 3 6 .4 

samples of student's workpieces 84 5.8 

others 18 1.2 



Type of items used frequently per subject area (SA): 
(in brackets: frequency of mentioning - cf . Table 69b) 

- SA 1: Education teacher trainings : 'short essay' (39), 
'short open-ended questions' (31)" , 'longer paper on a set 
subject' (28), and 'multiple choice' (23) (total number of 
nominations for this subject area: 162) 

- SA 2: Humanities : 'short essay' (35), 'short open-ended 
questions' (25), and 'longer paper on a set subject' (25) 
(total number of nominations for this subject area: 137) 

- SA 3: Languages^ linguistics : 'short essay' (43), 'short open- 
ended questions' (40)", 'multiple choice' (28), and 'longer 
paper on a set subject' (20) (total number of nominations for 
this subject area: 164) 

- SA 4: Law and legal professions : 'short essay' (24), 'longer 
paper on a set subject' (20), and 'short open-ended questions' 
(19) (total number of nominations for this subject area: 92) 

- SA 5: Economics :' short open-ended questions' (44), 'short 
essay' and 'multiple choice' (36 each) (total number of 
nominations for this subject area: 175) 

- SA 6: Agriculture : 'short open-ended questions' (12), 
'multiple choice' (11), and 'short essay' (9) (total number of 
nominations for this subject area: 50) 

- SA 7: Engineering and technical professions : 'short open-ended 
questions' (25), 'multiple choice' (21), 'short essay' (18); 
'questions to be filled-in', 'longer paper on a set subject', 
and 'workpieces' (11 each) (total number of nominations for 
this subject area: 104) 

- SA 8: Sciences : 'short open-ended questions' (31), 'multiple 
choice' (28), 'short essay' (21); 'questions to be filled-in' 
and 'longer paper on a set subject' (15 each) (total number of 
nominations for this subject area: 124) 
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3.5 Two-way communication: tutoring, counselling and assessment/'* 64 



- SA 9; Mathematics ; 'short open-ended questions' (27), 
'multiple choice' (26), 'questions tvo'be filled-in' (21), and 
'short essay' (14) (total number of nominations for this 
subject area: 109) 

- SA 10; Social s ciences : 'short essay' (43), 'short open-ended 
questions' (34), 'longer paper on a set subject' (34), and 
'multiple choice' (31) (total number of nominations for this 
subject area: 180) 

- SA 11: Medicine, medical Tobs : ^multiple choice' (13), 'short 
open-ended questions' (13), and 'short essay' (10) (total 
nu::nber of nominations for this subject area: 61 )• 
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3,5 Two-way communication: tutoring, counselling and assessment/ "65 



Table 69b: Type of items used for 


each 


subject area (L69): 




Subject area: 
( 1) education, teacher training 
( 2) humanities, music and arts 
( 3) languages, linguistics 
( 4) law and legal professions 
( 5) economics 

( 6) agriculture, sylviculture, forestry 
( 7) engineering and technical professions 
( 8) sciences 
( 9 ) mathematics 

(10) «ncial sciences 

(11) medicine and medical jobs 

(12) others 








Subject areas: 

(1) (2) (3) (4) (5) (6) 


(7) 


(8) 


(9) 


(10) 


(11) 


(12) Sum 


^ short open-ended questions 
31 25 40 19 44 12 


25 


31 


27 


34 


13 


20 


321 


multiple choice 
23 16 28 13 36 11 


21 


28 


26 


31 


13 


16 


262 


questions to be filled-in 
15 8 18 06 21 5 


11 


15 


21 


15 


7 


11 


153 


short essay on a set subject 
39 35 43 24 36 9 


18 


21 


14 


43 


10 


16 


308 


longer paper on a set ::ubject 
28 25 20 20 22 6 


11 


15 


9 


34 


7 


12 


209 


longer paper on self-chosen subject 
12 12 9 5 7 4 6 6 


4 


17 


6 


5 


93 


workpieces 
12 16 4 2 6 3 


11 


7 


7 


3 


4 


9 


84 


others 

2 0 2 3 3 0 


1 


1 


1 


3 


1 


1 


18 


Sum 

162 137 164 92 175 50 104 


124 


109 


180 


61 


90 


1448 
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3.5 Two-way conmuni cation : tutoring, counselling and assessment/ "66" 

Assignments submitted are not only corrected but also commented by 
the tutors at the great majority of institutions (Q46). 



Table 70: Tutor's comments on the assignments (Q46): 



- correction only: 

- correction and additional comments : 

- partly correction only, partly correction 
and additional comments: 



% related to N=185; 

12 X "no answer" for the question as a whole 



_J % 

12 6.5 
165 89.2 

. .8 4.3 



The comments on assignments by tutors are rather short (Q47): 



Table 71: Extent of tutor's comments on assignments 


(Q47) 


• 
• 






- about one page of the size of this report: 

- about 1/2 page of the size of this report: 

- less than half a page of the size of this report: 

- more than a page of the size of this report: 


f 
21 
68 
67 

6 


% 
13 
42 
41 

3. 


0 
0 
4 
7 




1% related to N=162; 35 x "no answer" 



Comments of tutors on assignments are usually indidualised (Q48): 
Table 72: (Non-) individualisation of tutor's comments (Q48): 



_^ % 

- individualised comments 135 75.8 

- non-individualised model answers 19 10.7 

- partly individualised, partly non-individualised.. 24 13.5 



|(% related to N=178; 19 x "no answer") 



The person correcting the assignments is also responsible for 
answering students' questions at the majority of institutions 
(Q49): 



3.5 Two-way conDniinication: tutoring, counselling and assessment/ -67- 



Table 73: Tutors' responsibility for: (Q49) 










yes 


no 








14 




- writing letters of encouragement or reminders . . 


. . 66 


112 








106 




(N=178; 19 X "no answer") 



The correctors /tutors are paid on a fee basis per corrected 
assignment in about half of the institutions (Q50): 



Table 74: Payment of correctors /tutors (Q50): 



yes no 

- on a fee basis, per corrected assignment 87 90 

- for all assignments in a course on a fee basis 25 152 

- a salary for employment 

- part time ; 64 113 

- full time 53 124 



(N=177; 20 x "no answer") 



Q51 refers to the use of a computer for the correction and 
commenting on assignments - separately for type of correction and 
subject area; the data on this question cannot be interpreted; 
most respondents have not differentiated between the subjects 
areas and/or the types of correction/commenting. 
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3.5.3 ^Learner Friendliness' or degree of learner support 

3.5.3.1 Introduction: the concept of learner friendliness/amount 
of support for learners 

The concept is related to M. Delling's term 'supporting 
organisation' (Celling 1971) for the distance-education 
institution and means the degree to which the institution gives 
support to the learners by means of tutoring and counselling and 
to which It is adaptive to students' needs and wishes (Holmberg & 
Schuemer 1985) . v y 

The learner-friendliness concept is based on the assumption that 
two-way communication between the learner and the institution is a 
central constitutive element of distance teaching beside the 
teaching material. This communication may have different forms and 
may be brought about by different methods - e.g. submission of 
assignments to be corrected and/or commented on by the 
institution, tutoring/counselling, face-to-face or on the 
telephone etc. -; but its aim should be to enable the learners to 
manage their studies and to overcome the difficulties in distance 
education. The communication should give the learner the feeling 
of not being alone. As different learners have different needs and 

- in addition - may react to the communication offers 
(media /methods) of the institutions in different ways a 
'supporting organisation' should not rely on one medium/method 
only. 

The concept is heuristic; it is hypothesised that the learners 
experience the distance-teaching institution as more friendly and 
that the institution can give more help to students 

- if two-way communication is considered a constitutive element of 
'the services offered by the institution and 

- the more components the communication includes (e.g. correction 
of/ commenting on the assignments submitted by the learners; 
mediated or "f ace-to-face contacts between tutor and learner; 
supplementary face-to-face sessions) 

- if the institution offers a counselling and tutoring service and 
this service uses different media/methods (e.g. written 
correspondence and/or telephone service and/or face to face 
contacts) and if this service is available also after usual 
office hours or on weekends 

- if the institution encourages learners to contact tutors when 
they feel they need help 

- if the institution contacts students who do not stay in touch 
(and the initiative to contact the institution is not left with 
the learners ) 

- if the institution takes steps to reduce non-starter and drop- 
out rates (e.g. mailing of standardised or individualised 
letters of encouragement or reminders if students have not been 
in touch for a certain time and/or phone calls to learners 

- if the main purpose of the assignments is learner support (and 
not merely assessment of learner performance) 

- if the learners may submit their assignments for correction and 
comment whenever it suits them 

- if the turn-around time for tutors' corrections and comments is 
not too long (e.g. does not exceed 9 days ) 
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- if the assignments submitted are not only corrected, but also 
commented on and if the tutor's comment is not too short 

- if the tutors/correctors receive a salai^r (and are not paid on a 
fee basis per corrected assignment) and, therefore, can spend as 
much time on each learner and her/his assignments as is 
necessary 

Obviously the list above is not complete (other elements - not 
mentioned - may be of relevance); furthermore not the mere 
existence of a two-way communication element or the mere quantity 
of such elements but the quality of the communication is decisive. 
But it is difficult - maybe impossible - to assess/measure quality 
by the crude method of a written survey/questionnaire. Therefore 
the operational definition above is only a rough indicator. 

The questionnaire contains several items related to the topics 
mentioned above. 

Each item can be transformed in such a way that it is scored one 
("1") if it points in the direction of "learner support' and is 
scored zero ("0") othezrwise^ > . Then the 'learner support ' -score 
(LF) can be taken simply as the sum of the 0/1-values over the 
items /conditions and represents the number of measures taken by 
the institution to support learners. The conditions and the 
corresponding items are summarised in Table 75. (The transformed 
items in Table 75 are designated by a preceding "T" . "S" denotes a 
transformed item based on a sum score.) 



Table 75: Definition of the LF -Score (LF: learner friendliness or 
degree of learner support) 



The LF-score increases, 

- if two-way communication is considered a constitutive element 
of the services offered by the institution (Q29) and if the 
two-way communication includes two or more of the 
following components (S29T2) 

— correction of assignments (T29a) 

— comsTients on the assignments (T29a) 

— regular contacts (mediated or face-to-face) between tutor 
and learner (T29c) 

— face-to-face sessions (T29d) 

— answering of lerners' requests (T29e) 

— contacts with learners initiated by the institution (T29f) 



Continued 



1) The if -then-else structure implies that 'non response' is 

scored as zero. The number of 'non-responses' per institution 
may be used for 'correction' (e.g. by means of partial 
correlation). If this number exceeds a certain limit to be 
specified the LF-score should not be interpreted. 
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Table 75 - continued 



- if the institution offers a counselling and tu:oring service 
(Q30) and this service includes at least two media /methods 
(S30T2) : 

— written correspondence (T30a) 

— telephone service (T30b) 

— audio tapes (T30c) 
face-to-face contacts (T30d) 

- if the service is available also after usual office hours or at 

week-ends (T31) 

- if the institution encourages learners to contact tutors when 
they feel they need help (T33) 

- if the institution contacts learners from whom nothing is heard 
(T34) ^ 

if the institution takes measures to reduce non-starter or 
drop-out rates - including at least two of the following 
measures (S35T2 ) : 

— mailing standardised letters of encouragement or reminders 
if students do not let heard from them for a certain time 
(T35a) 

— mailing individual letters (T35b) 

— mailing standardised or individualised encouraging audio 
tapes (T35c, d) . 

— phone calls to learners T35e) 

— visits to learners by people appointed by the institution 
(T35f ) 

- if the main purpose of the assignments is learner support (and 
not merely assessment of learner performance - T39) 

- if the learners may submit their assignments for correction and 
comment whenever it suits them (T43) 

- if the turn-around time for tutor's corrections and comments 
does not exceed 9 days (T44; the median of turn-around time lies 
between 9 and 10 days.) 

- if the assignments submitted are not only corrected, but also 
commented on (T46) 

- if the tutor's comment has the length of at least half a page of 
the size of this report (T47) 

- if the tutor's comment is individualised (T48) 

- if the tutors/correctors receive a salary (and are not paid 
on a fee basis per corrected assignment (T50) 



The LF-composite score is defined as^): 
LF= S29T2 T30 + S30T2 + T31 + T33 + T34 + S35T2 + T39 
T43 + T44 + T46 + T47 + T48 + T50 




2) Each of the element sums for Q29 (Q29i— Q29f), Q30 (Q30a--Q30d) 
and for Q35 (Q35a — Q35f) have been dichotomised; otherwise the 
sums are dependent on the number of elements (i.e.; the sums as 
components of LF would be weighted implicitly by the number of 
their elements; questions with only two possible answers would 
get a lower weight than questions with several possible 
responses) . 
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Preliminary analyses of the relationships between the conditions 
show low or moderate correlations between several of them (see 
section 3. 5 •3.2), but principal component or factor analyses of 
the items show that the structure of the variables contains more 
than one component or factor; so the homogeneity of the LF-score 
is rather low (see Table 76). Also a scalogram analysis according 
to Guttman yields a rather low coefficient of reproducibility 
(•76), 

[ Elimination of 'poor' items or splitting the LF-score into 
subscores may result in higher coefficients. A cluster analysis of 
the LF-items suggests two or three subscores. but further scale 
analyses are needed; they will be reported in a later paper. ] 



r 



Table 76: Item statistics for the LF-composite score 



item/ condition 


M 


s 


Tit 


ad justed^ > 

Tit 


ail 


S29T2 


.95 


.21 


.26 


. 18 


.27 


T30 


.89 


.31 


.40 


.29 


.60 


S30T2 


.80 


.40 


.51 


.38 


.70 


T31 


.32 


.47 


.36 


.18 


.38 


T33 


.88 


.33 


.41 


.29 


.58 


T34 


.61 


.49 


.43 


.24 


.56 


S35T2 


.50 


.50 


.56 


.40 


.38 


T39 


.69 


.47 


.38 


.20 


.29 


T43 


.59 


.49 


.33 


. 14 


.15 


T44 


.39 


.49 


.40 


.22 


.25 


T46 


.88 


.33 


.48 


. 36 


.43 


T47 


.48 


.50 


.39 


.20 


.31 


T48 


.69 


.47 ' 


.49 


.32 


.42 


T50 


.48 


.50 


.34 


.15 


.28 



Cronbach's alpha = {n/ (n-1 ) } {1- ( Ssx^'/si.^ ) } = ,61 

where n: number of items 

s±^t variance of item i 

s^^t variance of the composite score 

r^^: reliability (homogeneity) of the composite score 

- estimated by the Spearman-Brown formula from the 
mean item-intercorrelation (mean by using z') 

r^t = .63 



Coefficient of reproducibility: .76 



Distributuion of LF: 






range: 2-14 






median = 9; Qx 7; 


Q3 ^ 


11 


mean: 9,14; standard 


dev. J 


► 2.46 




3) adjusted r^^: item-score correlation adjusted for the part- 
whole effect by excluding item i from the sum before 
correlating 

4) coefficients loadings" ) for the first principal component 
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An example of an institution with a high LF-score is the Telford 
College in UK (LF=14); the Britsh Open University obtains a 
somewhat lower score (LF=11). The score of the German 

^^''''2''^^^^''^^^^^.^? (LF=7). very low scores are found at 

the Turkish Anadolu Universitesi (LF=3) and the Indonesian 
Universitas Terbuka (LF=2). 



3.5.3.2 Some relationships between LF-variables^ > 

The relationships are not presented here in detail (see Table A 2 
in Appendix 3) but some of the correlations between the LF- 
compcnents are described below (some other variables besides the 
0/1-items of the LF-score are considered): 

- The greater the number of elements of two-way communication fSum 
for Q29: S29) ^ 

-- the greater the nvimber of media in tutoring/counselling (Sum 
for Q30: written correspondence, telephone, audio tape, face- 
to face contact; S30): r=.39 

-- the greater the tendency to encourage learners to contact 
their tutors when they feel they need help (t33: rpb'-=.20) 

— the greater the number of measures taken by the institution 
to reduce the non-starter and drop-out . rates (Sum for Q35: 
S35): r= .30; and especially: 

— the greater the tendency to send individualised letters 
{T35b: rjpbi.,= .29) and use phone calls to learners (T35e: 
rB>fc,i.»= . 33 ) 

— the greater the tendency to use assignments for learner 
support (and not merely for assessment) (T39: rs,bi.. = . 26 ) . 

- The nximber of elements of two-way communication (Sum for Q29: 
S29) tends to be greater if the institution pays its correctors/ 
tutors a salary (T50: rpi,i..= .27) and does not pay them on a fee 
basis per corrected assignment 

- If the two-way communication includes correction of assignments 
(T29a) then it also tends to include commenting on assignments 
(T29b: phi=.74). ^ 

- Institutions which correct assignments (T29a) or comment on them 
(T29b) tend to answer learners' queries and requests (T29e; 

phi =.37 for T29a; phi =.46 for T29b). 

- If the two-way communication includes commenting on assignments 
(T29b) then the tendency is higher to describe Uearner support' 
as the main purpose of assignments (T39: phi=.28). 



5) We tried to study the possible effects of the sample 

heterogeneity with regard to the educational level of the 
institutions on the correlations by computing the coefficients 
only for those 72 institutions which can be* classified as 
universities. The coefficients for this subsample of 
institutions have the same tendency as the coefficients on the 
basis of the total sample (N=197). 

Furthermore the correlations may be biased by the scoring of 
"no answer" to an LF-item as zero; this possible effect was 
checked by computing the coefficients only for those 150 
institutions where the number of missing values is not greater 
than 4. These correlations also have the same tendency as those 
ER]C computed for the total sample. 
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- If the two-way communication includes regular contacts between 
tutor and learner (T29c2) or organised face-to-face sessions 
(T29d) then it is more likely that the counselling service can 
be contacted also beyond usual office hours (T31: phi=,36 for 
T29c2 and phi=,43 for T29d). 

- Institutions offering a tutoring/counselling service (T30) tend 
to encourage their learners to contact tutors when they feel 
they need help (T33: phi=,47). 

- If the tutoring service includes written correspondence (T30a) 
then telephone service tends to be offered too (T30b: phi=.49)- 

- Institutions which take measures to reduce the non-starter and 
drop-out rate (T35a,b) tend to contact students who do not stay 
in touch (T34: phi=.31 for T35a "mailing standardised letters of 
encouragement" and phi=.32 for T35b "mailing individualised 
letters" ) . 

- If the tutors get a salary rather than being paid on a fee basis 
(T50) then phone calls to learners (T35e: phi=.27) are more 
likely and the tendency to send standardised letters of 
encouragement is lower (T35a: phi=-.26), 

- The number of media used by the counselling/tutoring service 
(Sum of Q30: T30a: 'written correspondence'; T30b: 'telephone 
service'; T30c: 'audio tapes'; and T30d: 'face-to-face contact'; 
S30) tends to be higher 

— if the counselling/tutoring service can be contacted also 
after usual office hours and on week-ends (T31: r3ptoj..= .36) • 

— if the institutions encourage the learners to contact their 
tutors when they feel they need hexp (T33: r3ptei.= .40), 

— if the institutions write individualised letters (T35b; 
rjptoj.»=.22) and use phone calls to learners (T35e: rptoie=-26), 

— if a learner is assigned one personal tutor for all her/his 
courses CT36: rjptoj.,= .22) or for all problem areas 
(T37:r5>i.j..= .25) • 

- if the counselling/tutoring service can be contacted also after 
usual office hours and on week-ends (T31): 

the number of elements of two-way communicacion (Sum for Q29: 
S29) tends to be higher ( 

— the number of measures taken by the institution to reduce the 
non-starter or drop-out rate (Sum for Q35: S35) tends to be 
higher (r3ptoj.«=.21) . 

- If the institution encourages learners to contact their tutors 
when they feel they need help (T33) the number of measures taken 
by the institution to reduce the non-starter or drop-out rate 
(Sum for Q35: S35) tends to be higher (r3ptoi«=. 20 ) • 

- If the institution contacts learners who do not stay in touch 
(T34) the number of measures taken by the institution to reduce 
the non-starter or drop-out rate (Sum for Q35: S35) tends to be 
higher (rptoi.=. 39 ) . 

- The greater the number of measures taken by the institution to 
reduce the non-starter or drop-out rate (Sum for Q35: S3b) the 
greater the number of media in tutoring/counselling (Sum for 
Q30: written correspondence, telephone, audio tape, face-to-face 
contact) : r=. 31 

- If the institution designates 'learner support' as the main 
purpose of assignments (T39) qa 
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— the number of elements of two-way communication (Sum for 029: 
S29) tends to be higher ( rpt,i.= . 26 ) . 

— the tendency not only to correct but also to comment on 
assignments is greater (T46: phi=.28). 

non ?nSff."S^ ?° ^i""® individualised comments (and not only 
non-individualised model answers) is greater (T48: phi=.20). 

var^b?pf n^.''°''''?^2''i°"' ^^^r^" LF-score and some other 
variables not included m Table 76 are summarised in Table 77. 



Table 77: Correlations between LF-score and some other items 
(not included in Table 76) 



LF 

Q22b curriculum is fixed, but it contains alternatives 723 

Q27 encouragement of self help groups 12 

T29 two-way comm. as a constitutive element.*.*.*.*.* 18 

T29a correction of assignments .*!.*.*.*.'!.* .*40 

T29b commenting on assignments ] 47 

T29c regular contact between learner and* tutor: 
T29cl mediated 

T29c2 direct/face-to-face !*.*!!.*.*.*.*.*.*.*!.* 17 

T29d organised face-to-face sessions 14 

T29e answering of learners' queries ] ] .* ] 31 

T29f contacts with learners initiated by th4 *iiistitutioA * * .*23 

b^y number of elements in two-way communication 45 

S29N like S29 (without T29g) 

S29T2 0/1 score; "1": if S29N>2; else "0" '.26 

T30 offering of a counselling/tutoring service 40 

Medium of service ; 

T30a written correspondence 47 

T30al standardised letters .*!.*.*.*.*.*.*!.* 

T30a2 individualised letters .*.*.*.*.* 35 

T30a3 use of text modules ... .Ill . 1 1 1 06 

S30a (Sum of T30al/2/3) 52 

T30b telephone service co 

T30c audio tapes llllllllllllllllllll 15 

T30d face-to-face contact I .lllllllllllllllllll . * 25 

number of media for counselling (Sum of T30a/b/c/d) . . .53 

T3i counselling service after usual office hours or 
on week-ends 



S30 



36 
41 



T33 encouraging of learners to contact their tutors 

T34 contacting of learners who do not let hear from them .43 

- Continued - 
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Table 77: - Continued - 



S33_34 (sum of T33 and T34) 57 

Measures to reduce non-starter/drop-out rates (Q35): 

T35a mailing of standardised letters of encouragement 27 

T35b mailing of individualised letters 46 

T35c mailing of standardised audio tapes .08 

T35d mailing of individualised audio tapes 14 

T35e phone calls to learners 49 

T35f visits to learners 24 

S35 number of measures taken (Sum of T35a/b/c/d/e/f /g) . . . .55 

S35N like S35 but without T35g 58 

S35T2 0/1-score; "1": if S35N>2; else "0" 56 

T36 common tutor for all courses 18 

T37 common tutor for all problems 20 



lERLCi 



S4 
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3.5.3.3 Relation of the LF-score to success and non-starter rates 

There is no external criterion for the validity of the LF-Score* 
but if the concept has any validity at all there should be some' 
correlations between LF (and its components) and the success rate 
and/or the non-starter rate. Unfortunately there are only two very 
rough indicators for the latter two variables available: Q53 
(success rate for the three courses with the highest number of 
enrolments) and Q54 (the analogous non-starter rate) - cf. 
section 3.2.10 

The correlations for the LF-score and its components with Q53 and 
Q54 are summarised in Table 78 and those for some additional 
variables in Table 79; Table 80 contains corresponding 
crosstabulations for some of the relationships. 

The success rate (053: average percentage for the three courses 
with the highest number of enrolments) tends to be higher 

- the higher the LF-score is (LF: r=.26; n=139); the institutions 
with higher LF-scores (LF>10) tend to have the higher success 
rates (Q53>67) - cf . Table 78 and 80 

- if the institution uses ai: least two media /methods for 
counselling/tutoring (S30T2: rpt,j..= .23; n=139) 

- if the counselling service can be contacted by telephone also on 
week-ends (Q31cl: r=.22; n=124) 

- if the institution tries to reduce non-starter and drop-out rate 
by sending individualised letters (T35b: r=.39; n=139) or by 
phone calls to learners {T35e: r=.25; n=139) 

- if the institution takes at least two measures to reduce the 
non-starter and drop-out rates (S35T2: rpbia=.35; n=139) 

- if the tutors /correctors get a salary for full-time employment 
(Q50c2: r=.19; n=139) - see Table 79. 

- if the institution encourages the formation of self-help groups 
(Q27: r=.20; n=135) - see Table 79. 



The non-starter rate (054; average percentage for the three 
courses with the highest number of enrolments ) tends to be lower 

- the higher the LF-score (LF: r=-.16; n=121) 

- if the tutoring/counselling service includes individualised 
letters (Q30a2: r=-.30; n=96) 

- if the counselling service can be contacted by telephone on 
week-ends, too (Q31cl: r=-.26; n=109) 

- if the institution tries to reduce non-starter ^ua drop-out rate 
by sending individualised letters (T35b: r=-.30; n=121) 

- if the turn-around time for correction of assignments does not 
exceed 9 days (T44: r=-17; n=121) 

- if the tutors /correctors get a salary for full-time employment 
(Q50c2: r=-.25; n=llO) - see Table 79. 

- if the institution encourages the formation of self-help groups 
(Q27: r=-.17; n=114) - see Table 79. 
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Table 78: Correlation between the LF-score (and its components) 
and the success rate (Q53) and the non-starter-rate 
(Q54) 



N for Q53: 139; N for Q54: 121 



S29T2 at least 2 elements of two-way communication . . , 
T30 offering of a counselling/tutoring service . . . . , 

S30T2 at least two media for counselling 

T31 counselling service after usual office hours or 

on week-ends 

T33 encouraging learners to contact their tutors.... 



T34 
S35T2 

T39 
T43 

T44 

T46 

T47a 

T48 

T50 



053 054 



.06 
.04 
. 23 

.08 
.07 



contacting learners who do not stay in touch 14 

at least 2 measures to reduce non-starter/ 

drop-out rates (Q35) 35 

main purpose of assignments: learner support... 
submission of assignments whenever it suits the 

learner 

average turn-around time for correction: < 9 days 

also co/timenting on assignments , 

length of comments: half a page or more 

individualisation of comments 11 

payment of correctors /tutors : 
salary (full-time or part-time) 



-.13 

-.02 
.09 

08 
07 



-.06 
.06 
.03 

-.10 
.01 

-.10 

-.24 
,13 

.08 
-.17 
-.13 
-.08 
-.03 



.19 -.16 




These coefficients are rather low; but they give some support wO 
the assvimption that 'learner friendliness' or some of its 
components have some effect on the success rate and/or the non- 
starter rate. This seems to be true especially for 'individualised 
letters' (T35b; Q31a2) and 'phone calls to learners' (T35e). 

The tendency of these results agrees with the findings of step 1 
of this study (see Holmberg & Schuemer 1985, section 4.2.2.1). 
There, too, the institutions with higher scores in 'learner 
friendliness' had the higher success rates, but the correlation 
was weak (The LF- scale was constructed in the same way but 
consisted of fewer items). 
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Table 79: Correlations for some additional variables with the 
success ra te (Q53) and the non-starter rate (Q54) 

n: nuit±)er of pairs of oberservations per correlation 



Q27 encouragement of the formation of self-help groups .20 -.17 

n= 133 116 

Q30a2 individualised letters for counselling/ 

tutoring 15 ..30 

n= 107 96 



Q3lcl counselling/tutoring service can be contacted on 

weekend by telephone .22 -.25 

n= 124 109 



Q50cl tutors get a salary for part-time employment .17 -.02 

n= 127 110 

Q50c2 tutors get a salary for full-time employment .16 -.25 

n= 127 110 



n for the following correlations: n= 139 121 



Measures to reduce non-starter/drop-out rates (Q35): 

T35a mailing standardised letters of encouragement -.02 .02 

T35b mailing individualised letters 39 -.30 

T35c mailing standardised audio tapes -.01 .15 

T35d mailing individualised audio tapes -.01 .10 

T35e phone calls to learners 25 -.14 

T35f visits to learners 12 -.01 

S35 number of measures taken (Sum of T35a/b/c/d/e/f /g) .27 -.13 

S35N like S35 but without T35g 29 -.14 



T36 personal tutor assigned to learner for all courses .22 -.20 
T37a personal tutor assigned to learner for all problems .08 -.19 



The correlations of T36 and T37 with Q54 are in accordance with 
the Rekkedal's results (1985) that a personal tutor/counsellor for 
several courses contributes to reducing the drop-out rate. 



ERIC 
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Table 80: Crosstabulation : success and non-starter rate 
(Q53, Q54) vs- selected variables 



Q53 and Q54 dichotomised near median: 

Q53- : Q53 < 67 % Q53+ : Q53 > 67 % 

Q54- : Q54 < 12 % Q54+ : Q54 > 12 % 



LF<9 




Q53- 


Q53+ 


Q54- 


Q54 + 




r 


42 


30 


27 


35 




LF>10 


a 


(58.3) 


(41.7) 


(43.6) 


(56.5) 






11 


40 


35 


24 








(40.3) 


(59.7) 


(59.3) 


(40.7) 








Q53- 


r\c, Ox 
J + 




Q54 + 




S30T2<1 


z 


17 


5 


11 


9 




S30T2>2 


a 


(77.3) 


(22.8) 


(55.0) 


(45.0) 




I. 




^ c 
0 J 


51 


50 






a 


(44.4) 


(55.6) 


(50.5) 


(49.5) 








Q53- 


Q53 + 


Q54- 


Q54 + 




Z 


A O 


O A 

24 


22 


33 




S35T2>2 




( o4 . 2 ) 


(35.8) 


(40.0) 


(60,0) 




f 
i. 






A f\ 

40 


26 






a 


(^5.1) 


(63.9) 


(60.6) 


(39.4) 








Q53- 










T35b=0 




Q53+ 


Q54- 


Q54+ 




f 


48 


26 


26 


38 




T35b=l 


% 


(64.9) 


(35.1) 


(40.6) 


(59.4) 




f 


21 


44 


36 


21 






% 


(32.3) 


(67.7) 


(63.2) 


(36.8) 








Q53- 


Q53+ 


Q54- 


Q54+ 




T35e=0 


f 


46 


34 


31 


38 




T35e=l 


% 


(57.5) 


(42.5) 


(44.9) 


(55.1) 




f 


23 


36 


31 


21 






% 


(39.0) 


(61.0) 


(59.6) 


(40.4) 





r 
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3.5.4 Summary for tutoring/counselling and assessment 



- 90% of the institutions (176 of 194; 3 x "no answer") offer a 
counselling and tutoring service (Q30). It includes telephone 
service and written correspondence at the great majority of 
institutions (145 x and 151 x, resp.; see Tab. l);"but also 
opportunities for counselling by face-to-face contacts are 
offered by more than an half of the institutions (116 x); 
tutoring/ counselling by audio tapes is offered less frequently 
(60 X). 

The written correspondence for tutoring contains individualised 
letters (127); standardised letters or text modules are used 
less frequently (71 and 32 x , resp.). 

The tutoring/counselling service can be contacted not only on 
weekdays during the usual office hours but beyond these hours, 
too, in some institutions (see Tab. 2). 

Almost all institutions (192 out of 197) consider two-way 
communication as a constitutive element of their services (Q29). 
In this context the correction of and commenting on assignments 
and the answering of learners 's queries are mentioned very 
often. The great majority of the institutions (180 from 197) use 
rhree or more of these elements. 

The majority of the institutions take some measures to reduce 
the non-starter and drop-out rates. The 'mailing of standardised 
letters of encouragement' is the measure taken most frequently. 
Audio tapes are used very rarely for this purpose. 
Most institutions organise their tutoring/counselling service in 
such a way that the learner has to contact different tutors if 
he is enrolled for several courses; only at a few institutions 
is the learner assigned one central tutor for all courses (Q36). 
Similarly at many institutions learners have to contact 
different tutors/counsellors for different problems (Q37). 
About two thirds of the institutions encourage their learners tc 
form study groups or self-help groups (Q27). 

Almost all institutions use some form of continuous assessment 
(Q38); 'written assignments' and to a lesser degree 'written 
intermediary exams' are mentioned very often. 

'Assessment of learners' performance' and 'learner support' are 
assignment purposes mentioned equally often. 

Students have to participate in final course examinations at 85% 
of the institutions (Q40). The examinations are written in most 
cases . 

The acquisition of course certificates and the final marks 
depend on the marks obtained in the final examination as well as 
on tho marks awarded for assignments in more than half of the 
institutions (Q41 ) . 

About two thirds of the institutions have assignments in every 
course unit (Q42). 

About two thirds of the institutions allow the learners to 
submit their assignments whenever it suits them (Q43). 
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3.5.4 Summary for tutoring/counselling and assessment 



The average turn-around time for correction of assignments (i.e. 
the time lag from the day a student's assignment arrives until 
It IS returned with the tutor's corrections and comments: Q44) 
varies considerably between the institutions; about one half of 
the institutions need 10 or more days; a turn-around time of 14 
days or less is kept by 71% of the institutions (Q44). 
Predominant types/formats of items (Q45) are ^short open-ended 
questions', ^short essay on a set subject', and ^multiple 
choice '-items. The type of items preferred depends on the 
subject area to some degree (Q45). 

Assignments submitted are not only corrected but also commented 
by the tutors at the great majority (^90%) of the institutions 
(Q46). 

Tutors' comments on assignments are rather short: about half a 
page of the size of this report or less ((Q47: 135 x) and 
usually individualised (Q48: 135 x). 

The person correcting the assignments is also responsible for 
answering students' questions at the majority of institutions 
(Q49): 

The correctors/tutors are paid on a fee basis per corrected 
assignment in about one half of the institutions (Q50). 

The 'learner-friendliness' concept tries to integrate some 
aspects of tutoring/counselling and assessment of learner's 
performance . 

A composite score - based on this concept - is defined to 
measure che degree of learner support by the institution or the 
'learner friendliness' (LF-score). 

This score and some of its components show some relationships to 
the success rate (Q53) and the non-starter rate (Q54): 
If the LF-score is low (LF<9) then the success rates (Q53>67) 
are lower than if the LF-score is high (LF>10), 

Some additional correlations for items related to the ^learner- 
friendliness ' -concept : 

- The success rate (Q53: average percentage for the chree 
courses with the highest number of enrollments) tends to be 
higher 

- if the institution encourages the formation of self-help 
groups (Q27) 

- if the counselling service can be contacted by telephone 
also on week-ends (Q31cl) 

- if the institution tries to reduce non-starter and drop-out 
rate by sending individualised letters (T35b) or by phone 
calls to learners (T35e) 

- if the number of measures to reduce the non-starter and 
drop-out rates is higher (Sum over components of Q35, S35) 

- if the tutors/correctors get a salary for full-time 
employment (Q50c2) 

- The non-starter rate (Q54: average percentage for the three 
courses with the highest nu:nber of enrollments) tends to be 
lower 

- if the institution encourages the formation of self-help 
groups (Q27) 

- if the tutoring/counselling service includes individualised 
letters (Q30a2) 

- if the counselling service can be contacted by telephone on 
week-ends, too (Q31cl) 

- - M 



3.5*4 Summary for tutoring/counselling and assessment /'•82** 



- if the institution tries to reduce non-starter and drop-out 
rate by sending individualised letters (T35b) 

- if the turn-around time for correction of assignments does 
not exceed 9 days (T44) 

- if the tutors/correctors get a salary' for full-time 
employment (Q50c2) 

These results agree with the findings of an earlier study 
(Holmberg & Schuemer 1985, section 4,2 •2.1) • 
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3.6 Some further relationships 
(R. Schuemer & M. Weingartz) 

This section deals with the relationships between the composite 
scores defined in 3.2, 3.3 and 3.5. The following composite scores 
are considered: 



" amount of face-to -face components in distance teaching . 
The score is based on Q8 (cf. section 3.2.4) and has a range 
from 1 to 4: 

"1": ^pure distance teaching' (Q8a) 

"2": distance teaching with a few of face-to-face elements 
(98b, c) 

"3": distance teaching and face-to-face teaching of equal 

importance (Q8d) . 
"4": ^face to face teaching with a supplementary distance 

teaching material' (Q8e) 
(Some institutions gave multiple responses to Q8; therefore, the 
score cannot be defined for all institutions: N=166). 

- amount of flexibility or number of options students have 
with regard to their studies (cf. section 3.2.5). The score is 
based on Q9 and haz a ■ 'nge rom 0 to 8 . 

- LF: -learner friendliness! ^cf. section 3.5.3.1). The score is 
to assess the degree of learner support, it is based on 
items/conditions from Q29 , Q30, Q31, Q33, Q34, C35, Q39, Q43, 
Q44, Q46, Q47, Q48, and Q50 and has a range from 0 to 14. 

- IND: Score of independence (cf. section 3.3.4.1). The score is 
to indicate the extent of support enabling the learner to keep 
up self -directed study activities, i.e. activities that do not 

'depend on the presence of a teacher. The score has a range from 
0 to 7 and is based on items from Q9, Q22/23, Q24/25, Q26 , Q28, 
Q34, and Q47. 

- IC: Score of individ ualized cnntrni (cf. 3.3.4.2). The score is 
to indicate the extent of the control that takes into account 
and activates learners' previous knowledge, individual 
characteristics and aims. The score has a ranga from 0 to 7 and 
is based on items from Q17, Q2S , Q30, Q36, Q46, Q48, and Q49. 



- GC: Score of goal-oriented control (cf. 3.3.4.3). The score is 
to indicate the extent of control exerted on the learner in 
order for him to reach a fixed aim (degree, diploma) as 
effectively as possible. The score has a range from 0 to 6 and 
is based on items from Q27, Q30a, Q33, Q35, Q38, and Q39. 




Expectations /hypotheses : 

- There should be positive correlations between the flexibility 
(FI) and the LF-, IND-, or IC-score; on the other hand a 
negative correlation is expected between the Fl-Score and the 
Pl-Score: institutions with higher proportions of face-to-face 
elements can offer fewer options to their students with regard 
to time/hours of study. 
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3.6 Some further relationships/ "84'' 



- A negative correlation is expected between goal-oriented control 
and independence; it is assumed that goal-oriented control 
frequently makes use of measures that may lead to a restriction 
of the individual and therefore in the long run do not really 
suppport independence but could actually counteract it. 

- No correlation is expected between the amount of face-to^face 
components in distance teaching (Pl-Score) and the LF-, IND-, 
IC- or GC-scores • 

- Positive correlations are expected between 'learner 
friendliness' (LF), ' independence '( IND ) , and 'individualized 
control' ( IC) • 

Table 82 summarizes the correlation coefficients (Since there is 
item overlapping between the scores - i.e. some items are used for 
definition of scores in the same or similar way thus producing 
dependency of measurement and overestimation of the correlation 
between them - correlation coefficients are given both without and 
with correction^) 



1) corrected correlations: correlations after eliminating those 
items from one of the scores correlated which are contained in 
identically or similar way in the other score; that means for: 

- LF/IND: eliminating T34 and T47a from LF 

- LF/IC: eliminating S29T2, S30T2, T46 and T48 from LF 

- LF/GC: eliminating S30T2, S35T2, T33 and T39 from LF 

- LF/FI: eliminating T43 from LF 

- IC/GC: eliminating Q30T from IC 

- IND/FI: eliminating R9b from IND. 
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Table 81: Correlations between some composite scores 



- FI: amount of flexibility 

- LF: ^learner friendliness' 

- IND: Score of independence 

- GC: Score of goal-oriented control 

- IC: Score of individualized control 

- PI: amount of face-to-face components 



FI LF IND GC IC 



LF .34* 

(.27-)- 
IND .45'^ .62a 

( .20*^) ( .47a) 
GC .19* .42a .30* 

(.33a) 

IC .26* .69a .37* .29* 

(•52a) (.19*) 
PI -.28* -.03 -.OS -.12 .03 



In brackets: correlations corrected for item overlapping 
(see text) 

a : p<.Ol ; 

n=197 for correlations between FI, LF, IND, GC und IC 
n=166 for correlations of Pi with the other scores 



The empirical correlations ^ ) agree with the expectations - but 
there is one exception: contrary to the expectation there is a 
positive correlation between 'independence' and 'goal-oriented 
control ' . 



A factor analysis of the variables (with equamax rotation) shows 
that their relationships can be described by three factors^ > 
(explaining about 77 % of total variation): 

Factor I; high loadings (>.75) for LF, IND and IC; medium loading 
(.51) for FI. 

Therefore, the factor may be named/designated as 
'individualized learner support' 
Factor II: high positive loading (.86) for the Pl-score 

(proportion of face-to-face contacts) and medium negative 
loading (-.68) for 'flexibility' (FI). The factor may be 
interprpted as 'presence factor'. 



2) Pearson's r; the tendency of the coefficients does not change 
by computing Kendall's tau or Spearman's rho instead of 
Pearson's r. A further check was done by computing partial 
correlations with the number of missing values partial led out; 
The tendency of these partial correlations also agrees with the 
simple r-coef f icients . 

3) Factor scores for these three factors may be used for 
classifying the institutions by means of clusteranalysis . Such 
analyses will he described in a forthcoming paper. 
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Factor III ; high loading (.94) only for 'goal-oriented control'- 
The factor may be interpreted as 'general (non- 
individualized) control factor' . 



The negative correlation between flexibility (FI) and amount of 
face-to-face contatcs (PI) can also be shown by considering the 
means of FI for the four levels of PI (see Table 88): The highest 
mean of FI is found at PI=1 (pure distance teaching), the lowest 
for PI=4 (face-to-face teaching with supplementary distance 
teaching) . 



Table 82: Means 


of the flexibility score (FI) - depending on 




amount of face-to-face contacts (PI): 




(1) 


pure distance teaching (Q8a) 




(2) 


distance teaching with a few of face-to-face 






elements (Q8b,c) 




(3) 


distance teaching and face-to-face teaching of 






equal importance (Q8d) 




(4) 


face to face teaching with a supplementary 






distance teaching material' (Q8e) 




PI=1 


PI=2 PI=3 PI=4 




3.64 


2.94 2.29 1.57 F=4.76 p<.005 




(n) (39) 


(99) (14) (14) df= 3; 162 





Relationships between the composite scores and the success rate, 
the non-starter rate 

Table 83 shows the relationships between the composite scores 
mentioned above on the one hand and the success rate (Q53) and the 
non-starter rate (Q54) on the other hand. 



Table 


83: 


Correlations between the composite scores (FI, LF, IND, 
GC, IC, PI - see Table 82) and: 

- the success rate (Q53) 

- the non-starter rate (Q54) 






FI LF 


IND 


GC 


IC 


PI 


Q53 
Q54 


(n) 
(n) 


.04 .26* 

(139) (139) 

.07 -.16= - 
(121) (121) 


.19*= 
(139) 
.19*= 
(121) 


.07 
(139) 
-.01 

(121) 


.24* 

(139) 
-.08 
(121) 


. 17 = 

(119) 
-.16 
(105) 


a: p< 


.01 


b: p<.05 


c : p< . 


10 







The success rates (Q53) are higher with rising 'learner 
friendliness' (LP), 'score of independence' (IND), and 
'individualized control' (IC); the non-starter rates (Q54) are 
lower with higher scores in 'learner friendliness' or 
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' independence ' . 

Multiple regression/correlations with the scores as predictors and 
the success rate or the non-starter rate as dependent variable 
yield multiple correlation coefficients (R) not much higher than 
the highest coefficient (r) for the simple relationships: 

- R=.34 for the multiple correlation between FI, LF, IC, GC and 
IND as independent and Q53 as the dependent variable versus 
r=.26 for LF/Q53 (R=.35 when PI is used as an additional 
predictor) 

- R=,24 for the niultiple correlation between FI, LF, IC, GC and 
IND as independent and Q54 as the dependent variable versus 
r=-.19 for IND/Q54 (R=.28 when PI is included) 



The scores and type of ownership and educational level of 
institution 



Table 84(a,b) show the relationships between the scores and the 
type of ownership of the institution (Q2T) and the educational 
level (Q5/6): 



Table 84a: Correlations between the composite scores (FI, LF, IND, [ 
GC, IC, PI - see Table 82) and: 

- type of ownership (Q2T: "1" state owned; 
otherwise* "0" ) 

- educational level (Q5/6: "1" university study and 
further professional training and courses of this 
level for at least two subject areas; otherwise "0" 







Fl 


LF 


IND 


GC 


IC 


PI 


Q2T 




-.23 


-.06 


-.03 


-.15 


.07 


.04 


Q5/6 




-.01 


-.06 


-.no 


.16 


.02 


.02 



exclusive of the 11 institutions with multiple response to Q2 
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